Music generation method, electronic device, storage medium and computer program product

Abstract

The embodiment of the invention discloses a music generation method, electronic equipment, a storage medium and a computer program product. The method comprises the following steps: performing feature extraction on voice information of a target object to obtain voice semantic features and voice acoustic features of the target object; generating a human voice audio based on the voice semantic features and the voice acoustic features; based on the voice audio, generating a composing and music-making audio; and generating target music based on the voice audio and the arrangement music audio.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a music generation method, electronic device, storage medium and computer program product. Background Technology

[0002] When creating music (such as songs with lyrics), related technologies can employ both traditional and deep learning-based methods. Traditional methods require close collaboration between composers, lyricists, singers, and arrangers to complete a song, while deep learning-based methods generate songs based on user-input prompts. However, both methods suffer from low efficiency and limited flexibility. Summary of the Invention

[0003] To address the technical problems existing in related technologies, embodiments of this application provide a music generation method, an electronic device, a storage medium, and a computer program product.

[0004] To achieve the above objectives, the technical solution of this application embodiment is implemented as follows: In a first aspect, embodiments of this application provide a music generation method, the method comprising: Feature extraction is performed on the speech information of the target object to obtain the speech semantic features and speech acoustic features of the target object; Based on the speech semantic features and the speech acoustic features, generate human voice audio; Based on the human voice audio, generate the arranged background music audio; The target music is generated based on the human voice audio and the arranged background music audio.

[0005] Secondly, embodiments of this application also provide an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor; When the processor runs the computer program, it executes the steps of the music generation method described in the embodiments of this application.

[0006] Thirdly, embodiments of this application also provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the music generation method described in embodiments of this application.

[0007] Fourthly, embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the music generation method described in embodiments of this application.

[0008] The music generation method, electronic device, storage medium, and computer program product provided in this application extract features from the speech information of a target object to obtain the speech semantic features and speech acoustic features of the target object; based on the speech semantic features and the speech acoustic features, a human voice audio is generated; based on the human voice audio, an arranged music audio is generated; and based on the human voice audio and the arranged music audio, target music is generated. By adopting the technical solution of this application embodiment, after obtaining the speech semantic features and speech acoustic features of the target object by performing semantic understanding and acoustic feature extraction on the speech information input by the target object, target music conforming to the speech semantic and speech acoustic features is generated. The target object does not need to go through a complex production process; it only needs to input speech information to quickly create personalized music, achieving the goal of creating a song from a single sentence, thus improving the efficiency and flexibility of music creation. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating the music generation method according to an embodiment of this application. Figure 1 ; Figure 2 This is a flowchart illustrating the music generation method according to an embodiment of this application. Figure 2 ; Figure 3 This is a flowchart illustrating the music generation method according to an embodiment of this application. Figure 3 ; Figure 4 This is a flowchart illustrating the music generation method according to an embodiment of this application. Figure 4 ; Figure 5 This is a schematic diagram of the model architecture of the music generation method according to an embodiment of this application; Figure 6 This is a schematic diagram of the lyrics generation process according to an embodiment of this application; Figure 7 This is a schematic diagram of the vocal score generation process according to an embodiment of this application; Figure 8 This is a schematic diagram illustrating the music arrangement and composition generation process in an embodiment of this application. Figure 9 This is a schematic diagram of the composition of the music generation device according to an embodiment of this application; Figure 10 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0010] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0011] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0012] When creating songs, related technologies typically employ both traditional composition methods and deep learning-based methods. Traditional composition methods require close collaboration between composers, lyricists, singers, and arrangers to complete a song. This demands not only professional musical knowledge and skills but also a significant investment of time and resources, placing high demands on users and making it unsuitable for general, non-professional users.

[0013] In song creation using deep learning-based methods, users input text (i.e., prompt text) to generate corresponding audio recordings (also known as song audio, or simply songs). For example, current commercial products from Suno and Tiangong use deep learning to generate corresponding audio recordings based on user-input prompt text. This audio recording includes lyrics, melody, vocals, and arrangement. Voice cloning can be used to enable a fixed user to sing a song; for example, QQ Music's AI-assisted singing feature can clone a user's voice to perform a song. The drawbacks of this song audio creation method are that inputting all the lyrics takes a considerable amount of time, is cumbersome, and results in low song creation efficiency. Furthermore, it can only generate lyrics identical to the user's input text and can only synthesize the vocal parts based on fixed (or random) vocal sources, failing to incorporate the user as the singer into the audio recording, thus limiting the flexibility of song creation. Furthermore, after cloning a user's voice, songs that can be synthesized for singing need to be specially produced. The process includes: accompaniment production, lyrics and electronic score production via Musical Instrument Digital Interface (MIDI). The production cost is high, the production process is complex, and only songs with copyright information can be generated, which severely restricts the user's playability.

[0014] This shows that the song creation methods in related technologies suffer from low creation efficiency and poor flexibility.

[0015] Based on this, this application proposes a music generation method. In various embodiments of this application, by performing semantic understanding and acoustic feature extraction on the voice information input by the target object, the target object's voice semantic features and voice acoustic features are obtained, and target music that conforms to the voice semantic and voice acoustic features is generated. The target object does not need to go through a complicated production process. It can quickly create music with personalized characteristics by simply inputting voice information, achieving the goal of creating a song with just one sentence, and improving the efficiency and flexibility of music creation.

[0016] This application provides a music generation method, which is applied to an electronic device. Figure 1 This is a flowchart illustrating the music generation method according to an embodiment of this application. Figure 1 ,like Figure 1 As shown, the music generation method includes: Step 101: Extract features from the speech information of the target object to obtain the speech semantic features and speech acoustic features of the target object.

[0017] In this embodiment, the music generation method can be applied to multiple fields such as music production, audiobooks, short video creation, metaverse entertainment, and educational applications. The electronic device can refer to a device with music generation capabilities, such as a music generator or a smartphone or computer containing music generation tools (e.g., music production software).

[0018] In one embodiment, before extracting features from the speech information of the target object to obtain its speech semantic features and speech acoustic features, the method further includes: acquiring the speech information of the target object. This speech information can be input by the target object (i.e., the user) and acquired through a data acquisition device installed in an electronic device.

[0019] Here, the target object can be a user related to the generated music, and the target object's voice information can be understood as descriptive voice audio input by the target object, such as: "Please help me generate a song that expresses love for the starry sky, and is quite childlike and cheerful." The voice information can also be called voice audio, voice prompt information, or voice prompts; this embodiment does not limit the name of the voice information.

[0020] Here, the music can be a piece of music without lyrics or a song with lyrics. In this embodiment, the music generated is for a song with lyrics.

[0021] In practical applications, electronic devices can perform semantic understanding on the speech information input by the target object to obtain speech semantic features. This semantic understanding can also be called semantic recognition. Specifically, an Automatic Speech Recognition (ASR) module can be used to extract textual semantics from the speech information input by the target object to obtain the corresponding speech semantic features. In practical applications, electronic devices can use specialized audio coding models, such as the Hidden Unit BERT (HUBERT) module, to extract features from the speech information input by the target object to obtain speech acoustic features.

[0022] It should be noted that speech acoustic features are audio features that are independent of semantics. Speech acoustic features may include, for example, timbre, voiceprint, and other features that can represent the characteristics of the target object's voice. In other words, speech acoustic features include timbre features and voiceprint features.

[0023] Step 102: Generate human voice audio based on the speech semantic features and the speech acoustic features.

[0024] In this embodiment of the application, the human voice audio may also be referred to as synthesized human voice singing audio, or as human voice audio sung by the user, or as user vocal audio. This embodiment of the application does not limit the name of the human voice audio.

[0025] In practical applications, in one embodiment, generating human voice audio based on the speech semantic features and the speech acoustic features includes: Based on the aforementioned speech and semantic features, lyrics text information and vocal scores are generated; The human voice audio is generated based on the lyrics text information, the vocal score, and the speech acoustic features.

[0026] Here, the lyrics text information can also be called song lyrics, lyrics, structured lyrics, structured lyrics text information, or lyrics data. This embodiment of the application does not limit the name of the lyrics text information. The vocal score can also be called vocal melody, vocal main melody, arranged melody, musical note MIDI score, MIDI score, or song score. This embodiment of the application does not limit the name of the vocal score.

[0027] In practical applications, electronic devices perform semantic understanding on the voice information input by the target object and generate lyrics text information that conforms to the voice semantics of the target object based on the user's voice semantics (i.e., voice semantic features).

[0028] Based on this, in one embodiment, generating lyrics text information based on the speech semantic features includes: Emotion recognition is performed on the speech information of the target object to obtain the speech emotion features of the target object; The lyrics text information is generated based on the speech semantic features and the speech emotional features.

[0029] Here, the voice emotion features can also be called voice mood features, which may include, for example, features such as joy, sadness, and excitement.

[0030] Here, the electronic device can perform emotion classification training on the speech information input by the target object, and can identify the emotional information in the speech information, that is, obtain the speech emotion features of the target object. Specifically, the electronic device can use a Transformer model + fully connected layer to perform speech emotion classification training on the speech information.

[0031] In one embodiment, generating the lyrics text information based on the speech semantic features and the speech emotion features includes: encoding the speech semantic features to obtain a speech semantic vector, and encoding the speech emotion features to obtain a speech emotion vector; inputting the speech semantic vector and the speech emotion vector into a lyrics generation model to obtain the lyrics text information output by the lyrics generation model. In this embodiment, the lyrics generation model can be composed of a long sequence Transformer (Transformer XL, Transformer Extra Long) model.

[0032] Here, in one embodiment, before inputting the speech semantic vector and the speech emotion vector into the lyrics generation model to obtain the lyrics text information output by the lyrics generation model, the method further includes: generating the lyrics generation model. Specifically, generating the lyrics generation model includes: acquiring training data; the training data includes lyrics data and a melody synchronized with the lyrics data; constructing a target loss function; and generating the lyrics generation model based on the training data and the target loss function.

[0033] Here, the training data is a pairing dataset, which consists of lyrics data and their corresponding melody pairing data. In practical applications, electronic devices can collect datasets containing a large number of lyrics from online music platforms, lyrics databases, etc., and clean the collected datasets to remove information irrelevant to the lyrics, retaining pure lyrics data. Next, lyrics data synchronized with the melody are collected from the pure lyrics data to ensure precise temporal alignment between the lyrics and the melody.

[0034] In practical applications, the target loss function can be constructed using the following formula (1): (1).

[0035] in, This represents the target loss function, which is the loss function defined for lyrics generation; Indicates the number of words in the lyrics sequence; Indicates the first in the lyric sequence One word; This indicates that the given preceding text (i.e., the lyric sequence includes the 0th word) ),…,No. Words ( In the case of the first The probability of a word, the condition for which the probability is the speech semantic vector. and voice emotion vectors .

[0036] In practical applications, electronic devices can combine user voice emotional information to create vocal scores, making the generated vocal scores more in line with the user's current emotions.

[0037] Based on this, in one embodiment, generating a human voice score based on the speech semantic features includes: Emotion recognition is performed on the speech information of the target object to obtain the speech emotion features of the target object; The vocal score is generated based on the speech semantic features, the lyrics text information, and the speech emotional features.

[0038] In one embodiment, generating the vocal score based on the speech semantic features, the lyrics text information, and the speech emotion features includes: encoding the speech semantic features to obtain a speech semantic vector, encoding the speech emotion features to obtain a speech emotion vector, and encoding the lyrics text information to obtain a lyrics vector; inputting the speech semantic vector, the speech emotion vector, and the lyrics vector into a vocal score generation model to obtain the vocal score output by the vocal score generation model.

[0039] It should be noted that the vocal score generation model is similar to the lyrics generation model described above and can be constructed using a Transformer XL model. The vocal score output by the vocal score generation model can be a MIDI score corresponding to the lyrics and conforming to the standards of speech emotion vectors and speech semantic vectors.

[0040] In practical applications, when creating vocal scores, electronic devices can also combine user listening preference data to create vocal scores, making the generated vocal scores more in line with the user's listening habits. In this way, by incorporating user preferences, the generated music can be more easily accepted by users and meet their personalized creative needs.

[0041] Based on this, in one embodiment, generating the vocal score based on the speech semantic features, the lyrics text information, and the speech emotional features includes: Obtain the preference data of the target object; The vocal score is generated based on the speech semantic features, the lyrics text information, the speech emotional features, and the target object's preference data.

[0042] Here, the target object's preference data can also be referred to as the target object's music listening preference data, or as the target object's music preference description information. This application embodiment does not limit the name of the preference data. The target object's preference data may include the target object's preferred style and preferred rhythm.

[0043] In one embodiment, generating the vocal score based on the speech semantic features, the lyrics text information, the speech emotion features, and the target object's preference data includes: encoding the speech semantic features to obtain a speech semantic vector; encoding the speech emotion features to obtain a speech emotion vector; encoding the lyrics text information to obtain a lyrics vector; and encoding the target object's preference data to obtain a preference reference vector; inputting the speech semantic vector, the speech emotion vector, the lyrics vector, and the preference reference vector into a vocal score generation model to obtain the vocal score output by the vocal score generation model.

[0044] Here, when creating vocal scores, electronic devices can consider not only the user's emotional information in their voice, but also the user's listening preferences. This means that the user's listening preferences are also used as input to the vocal score generation model, constraining the generated scores to better suit the user's listening habits. Specifically, taking the target audience's preference data as a reference track as an example, the electronic device first collects the user's preferred reference tracks. Then, it uses an audio encoder (AE) to obtain corresponding preference reference vectors from these tracks. These vectors are then input into the vocal score generation model and fused with the aforementioned features (i.e., speech semantic vectors, speech emotion vectors, and lyrics vectors). By referencing the user's preferred listening style and rhythm in the generated vocal score, the model can meet the user's personalized creative needs.

[0045] Step 103: Generate background music audio based on the human voice audio.

[0046] Here, the arranged music audio can also be called arranged music audio, or music arrangement, or arrangement audio. This application embodiment does not limit the name of the arranged music audio.

[0047] In practical application, in one embodiment, generating the arranged background music audio based on the human voice audio includes: Emotion recognition is performed on the speech information of the target object to obtain the speech emotion features of the target object; The musical arrangement and background music are generated based on the speech semantic features, the human voice audio, and the speech emotional features.

[0048] In one embodiment, generating the music arrangement audio based on the speech semantic features, the human voice audio, and the speech emotion features includes: encoding the speech semantic features to obtain a speech semantic vector; encoding the speech emotion features to obtain a speech emotion vector; and inputting the speech semantic vector, the speech emotion vector, and the human voice audio into the music arrangement generation model to obtain the music arrangement audio output by the music arrangement generation model.

[0049] It should be noted that the music arrangement and background music generation model can be composed of a Transformer XL model or a diffusion acoustic model, and this embodiment of the application does not limit it.

[0050] In practical applications, when creating music and background music audio, electronic devices can also combine user listening preference data to create music and background music audio that matches the user's listening habits.

[0051] Based on this, in one embodiment, generating the music arrangement audio based on the speech semantic features, the human voice audio, and the speech emotional features includes: Obtain the preference data of the target object; The music arrangement and background music are generated based on the speech semantic features, the human voice audio, the speech emotional features, and the target object's preference data.

[0052] In one embodiment, generating the music arrangement audio based on the speech semantic features, the human voice audio, the speech emotion features, and the target object's preference data includes: encoding the speech semantic features to obtain a speech semantic vector, encoding the speech emotion features to obtain a speech emotion vector, and encoding the target object's preference data to obtain a preference reference vector; inputting the speech semantic vector, the human voice audio, the speech emotion vector, and the preference reference vector into the music arrangement generation model to obtain the music arrangement audio output by the music arrangement generation model.

[0053] Here, when creating music and background music, electronic devices can consider not only the user's emotional information in their voice, but also the user's listening preferences. This means that the user's listening preferences are also used as input to the music and background music generation model, constraining the generated music and background music to better suit the user's listening habits. Specifically, taking the target audience's preference data as a reference track for user preferences as an example, first, the electronic device collects the user's preferred reference tracks. Then, it uses an audio processing unit (AE) to obtain the corresponding preference reference vector. This preference reference vector is input into the music and background music generation model and fused with the aforementioned features (i.e., voice semantic vector, human voice audio, and voice emotional vector). By referencing the user's preferred listening style and rhythm in the generated music and background music, the model can meet the user's personalized creative needs.

[0054] Step 104: Generate target music based on the human voice audio and the arranged background music audio.

[0055] In one embodiment, generating target music based on the human voice audio and the arranged background music audio includes: aligning the human voice audio and the arranged background music audio to generate the target music.

[0056] In one embodiment, after generating the target music based on the human voice audio and the arranged background music audio, the method further includes: outputting the target music. Specifically, the electronic device provides the generated target music to the target object for playback.

[0057] Here, the target music may also be referred to as the target song, or the target song audio, or the target song including lyrics. This application embodiment does not limit the name of the target music.

[0058] This application also provides another method for generating music, which is applied to electronic devices. Figure 2 This is a flowchart illustrating the music generation method according to an embodiment of this application. Figure 2 ,like Figure 2 As shown, the music generation method includes: Step 201: Obtain the voice information of the target object.

[0059] Step 202: Extract features from the speech information of the target object to obtain the speech semantic features and speech acoustic features of the target object.

[0060] Step 203: Based on the aforementioned speech semantic features, generate lyrics text information and vocal score.

[0061] In one embodiment, generating lyrics text information based on the speech semantic features includes: Emotion recognition is performed on the speech information of the target object to obtain the speech emotion features of the target object; The lyrics text information is generated based on the speech semantic features and the speech emotional features.

[0062] In one embodiment, generating a vocal score based on the speech semantic features includes: Emotion recognition is performed on the speech information of the target object to obtain the speech emotion features of the target object; The vocal score is generated based on the speech semantic features, the lyrics text information, and the speech emotional features.

[0063] In one embodiment, generating the vocal score based on the speech semantic features, the lyrics text information, and the speech emotional features includes: Obtain the preference data of the target object; The vocal score is generated based on the speech semantic features, the lyrics text information, the speech emotional features, and the target object's preference data.

[0064] Step 204: Generate human voice audio based on the lyrics text information, the vocal score, and the speech acoustic features.

[0065] Step 205: Perform emotion recognition on the speech information of the target object to obtain the speech emotion features of the target object, and generate an arranged music audio based on the speech semantic features, the human voice audio and the speech emotion features.

[0066] In one embodiment, generating the music arrangement audio based on the speech semantic features, the human voice audio, and the speech emotional features includes: Obtain the preference data of the target object; The music arrangement and background music are generated based on the speech semantic features, the human voice audio, the speech emotional features, and the target object's preference data.

[0067] Step 206: Generate target music based on the human voice audio and the arranged background music audio.

[0068] It should be noted that the specific processing steps for music generation by electronic devices have been detailed above and will not be repeated here.

[0069] By adopting the technical solution of this application embodiment, after semantic understanding and acoustic feature extraction of the voice information input by the target object, the target object's voice semantic features and voice acoustic features are obtained, and target music that conforms to the voice semantic and voice acoustic features is generated. The target object does not need to go through a complicated production process. It can quickly create music with personalized characteristics by simply inputting voice information, achieving the goal of creating a song with just one sentence, and improving the efficiency and flexibility of music creation.

[0070] The present application will be described below with reference to application examples.

[0071] To address the problems of low efficiency and poor flexibility in song creation in related technologies, this application provides a method for generating songs from user voice input. This method, based on the user's voice input (corresponding to the voice information of the aforementioned target object), performs semantic understanding and timbre feature extraction on the input user voice, generating song audio (corresponding to the aforementioned target music) that conforms to the user's voice semantics and acoustic features (corresponding to the aforementioned voice semantic features and voice acoustic features), thereby improving the efficiency and flexibility of song creation. The innovation of this application lies in: 1. By performing semantic understanding of user speech, the system generates song lyrics and arrangements that match the semantics of the user's speech. Users can quickly create personalized songs without going through complicated production processes, achieving the goal of creating songs from a single sentence. This improves the efficiency and flexibility of song creation, allowing users to explore and express their musical ideas more freely, meeting their personalized needs, and enhancing their song creation experience.

[0072] 2. By extracting the acoustic features of the user's voice, the user's acoustic features are used to complete the vocal part of the song. That is, the user can be placed as the singer in the song, making the user more immersed, improving the flexibility of song creation, and enhancing the user's song creation experience.

[0073] 3. This application can also determine the emotional information of the user's voice, and then combine the emotional information of the user's voice to create song arrangement (also known as song score, corresponding to the aforementioned vocal score), so that the generated song score can better match the user's current emotion.

[0074] 4. This application can also combine user listening preference data to create song arrangements, so that the generated song scores can better meet the user's listening habits. In this way, by integrating user preferences, the generated songs can be more easily accepted by users and meet the user's personalized creative needs.

[0075] This application proposes a method for generating songs from user voice input. This method has broad market application prospects and can be applied to multiple fields such as music production, audiobooks, short video creation, metaverse entertainment, and educational applications. This method can directly convert user voice input into personalized musical works, allowing users to directly participate in the song generation process through voice input. It provides a highly interactive and personalized experience and can also meet users' needs for personalized music content.

[0076] Figure 3 This is a flowchart illustrating the music generation method according to an embodiment of this application. Figure 3 ,like Figure 3 As shown, the user inputs voice information, for example, the user inputs "I want to generate a song that expresses love and freedom with a light rhythm." This voice information is semantically parsed to obtain the corresponding user semantic information, and the user voice feature is parsed, that is, the acoustic features of the voice information are extracted. After understanding the semantics and extracting the acoustic features, lyrics are generated based on the user semantic information. Based on the generated lyrics and user semantic information, a vocal melody (corresponding to the aforementioned vocal score) is generated, and user vocal audio (corresponding to the aforementioned vocal audio) is synthesized based on the lyrics, vocal melody, and acoustic features. Arrangement and backing audio (corresponding to the aforementioned arrangement and backing audio) is synthesized based on the user vocal audio. Finally, the user vocal audio and the arrangement and backing audio are aligned to generate a song audio that conforms to the user semantic information and acoustic features (corresponding to the aforementioned target music). This song audio includes the user's vocal performance based on acoustic features and the corresponding arrangement and backing audio. This solution allows users to generate a complete song audio by inputting a single voice message, lowering the barrier to entry for users while presenting the song as a whole (e.g., presenting the context, style, lyrics, composition, user singing, arrangement, and music).

[0077] In addition, this application can also combine the user's preferred listening style (collected or input, corresponding to the aforementioned target object's preference data), and by integrating user preferences, constrain the generated songs to be more easily accepted by users.

[0078] This application can facilitate the generation of intelligent audio content, allowing users to create copyrighted song audio content with a low barrier to entry. It can cover a large number of potential application areas such as audio drama production, converged media production, short video production, metaverse entertainment, product display media production, and educational applications.

[0079] Compared with related technologies, the solution proposed in this application not only extracts user timbre features (i.e., speech acoustic features) and generates vocal performances that match the user's timbre, but also generates corresponding accompaniment and arrangements based on semantic information (corresponding to the aforementioned vocal audio and arrangement audio). This means that users can quickly create songs with personalized characteristics without going through a complex production process. In addition, the solution proposed in this application provides users with greater creative freedom, free from the restrictions of existing copyright information, allowing users to explore and express their musical ideas more freely.

[0080] Figure 4 This is a flowchart illustrating the music generation method according to an embodiment of this application. Figure 4 ,like Figure 4 As shown, the process of this method generally includes the following steps: The user inputs descriptive speech audio (corresponding to the speech information of the aforementioned target object), and the speech audio is processed in the following two ways: semantic understanding and extraction of audio features; semantic-based generation of lyrics and semantic-based generation of vocal melody; generation of user-sung vocal audio based on the user's speech audio features, lyrics, and vocal melody; generation of arranged and matched audio based on the vocal audio of the vocal melody; finally, aligning the vocal audio of the vocal melody and the arranged and matched audio, and performing beautification post-processing, outputting a song audio with vocal performance conforming to the user's audio features and lyrics and music conforming to semantic information.

[0081] Figure 5 This is a schematic diagram of the model architecture of the music generation method according to an embodiment of this application. The method involves an integrated song synthesis scheme. This model architecture includes a lyrics generation module, a vocal score generation module, a vocal audio generation module, and an arrangement and background music audio generation module. The modules interact through the feature encoding of semantic features or audio features extracted from the user's input audio, ensuring that the final end-to-end generated song audio not only contains relevant information but also guarantees the stability of the generation. The following section combines... Figure 5 The music generation method of this application is described in detail, specifically including the following steps: Step 1: User Voice Analysis [Input]: User input voice prompts (i.e., user audio prompts), for example: "Please help me generate a song that expresses my love for the starry sky, something childlike and cheerful."

[0082] [Output]: Text information corresponding to the speech, text semantic vector, audio vector unrelated to semantics, and audio emotion classification vector.

[0083] 1. Use the ASR module to extract semantic text from user-input speech: Encode the extracted text information using the encoder of a pre-trained Transformer model (BERT, Bidirectional Encoder Representation from Transformers) to obtain speech semantic vectors. This speech semantic vector will guide the lyrics generation in step two and the music arrangement generation in step five, to ensure that the generated song content closely matches the user's semantic prompts. 2. Use the HUBERT module to extract semantically independent acoustic features from user-input audio: Use the HUBERT module to extract semantically independent user acoustic feature vectors from the speech. The user's acoustic feature vector will be used to guide the generation of vocal scores in step three below, so as to ensure that the synthesized song vocals are consistent with the user's timbre. Acoustic features include timbre, voiceprint and other relevant features that can express the characteristics of the sound. 3. By training speech to classify emotions, the emotional information (or mood information) within the speech can be identified, such as joy, sadness, and excitement. The information of the nodes (i.e., neurons) in the layer above the classification layer of the neural network contains emotional information, and the speech emotion vector of that layer's nodes... This will guide the generation of lyrics in step two, the generation of vocal scores in step three, and the generation of arrangement and music in step five, to ensure that the final synthesized song's emotion is consistent with the user's voice emotion.

[0084] Step 2: Lyrics Generation [Input]: Speech semantic vector Voice emotion vector .

[0085] [Output]: Structured lyrics, such as "Twinkle, twinkle, little star, how I wonder what you are."

[0086] Figure 6 This is a schematic diagram of the lyrics generation process according to an embodiment of this application, such as... Figure 6 As shown, the lyrics generation module receives the speech semantic vector from step 1. (For example, "songs that express a love for the starry sky, which are more childlike and cheerful") and voice emotion vectors (For example, "joyful"), input into the main model of the lyrics generation module, generates structured lyrics text information (e.g., "Twinkle, twinkle, little star, how I wonder what you are!"). The main model of the lyrics generation module can be composed of a Transformer XL model, which processes the speech semantic vectors... and voice emotion vectors The BERT model's encoder performs secondary encoding, and the Transformer XL's decoder decodes and outputs the corresponding lyrics.

[0087] The following describes the training process of the main model of the lyrics generation module.

[0088] 1. Training data preparation 1.1 Lyrics Dataset: We collect datasets containing a large number of lyrics from online music platforms and lyric databases. We then clean the collected lyric data, removing irrelevant information and retaining the pure lyric text.

[0089] 1.2 Lyrics-Melody Pairing Dataset: Collect lyrics data synchronized with the melody to ensure precise temporal alignment between the lyrics and melody. Use audio processing tools such as the Montreal Forced Aligner (MFA) to extract the fundamental melody from the audio file and synchronize it with the lyrics. In other words, the training data includes lyrics data and matching data between the lyrics and their corresponding melodies.

[0090] 2. Training loss function constraints The loss function for lyrics generation is defined by the following formula (1): (1).

[0091] in, Indicates the number of words in the lyrics sequence; Indicates the first in the lyric sequence One word; Given the preceding context, the first The probability of a word, given by the speech-semantic vector. and voice emotion vectors .

[0092] 3. Generate the main model of the lyrics generation module based on the training data and loss function constraints obtained above.

[0093] Step 3: Generating Vocal Scores [Input]: Voice emotion vector (For example, it contains children's voices and cheerful information in the sound), speech semantic vectors (For example, the information "relatively childlike and cheerful"), and the generated structured lyrics (for example, "Twinkle, twinkle, little star" generated in the previous step).

[0094] [Output]: MIDI sheet music (i.e., vocal sheet music) that corresponds to the lyrics and conforms to the standards of speech emotion vector and speech semantic vector.

[0095] 1. The main model of the vocal score generation module is similar to the main model of the lyrics generation module in step two. It can also be constructed using the Transformer XL model, which is used to process the speech emotion vector. Speech semantic vectors The lyrics are encoded a second time using the BERT model's encoder, and then decoded and output using the Transformer XL's decoder to obtain the corresponding MIDI score. 2. The training dataset uses the lyrics-melody pairing dataset collected in step 2, section 3.2; 3. The overall supervised loss function is the same as in step two, except that it does not predict the text itself. The prediction sequence format is: note|time point|duration, for example, "F4|3200|680" means that the pitch of F4 lasts for 680 milliseconds at 3200 milliseconds. In this application, the melody corresponding to the lyrics can be directly generated from the model using steps 1-3 above. The model generates the corresponding melody based on the timbre and emotion of the prompt speech and the semantics of the text using a probability model. Furthermore, user listening preference data can be incorporated as input conditions to the model, constraining the generated arrangement melody to better suit the user's listening habits. Figure 7 This is a schematic diagram of the vocal score generation process according to an embodiment of this application, such as... Figure 7 As shown, the collected user preference reference songs are processed by an Audio Encoder to obtain corresponding reference vectors. The data is fed into the Transformer XL model via cross attention, and then compared with the aforementioned features (i.e., speech semantic vectors). (For example, "songs that express a love for the starry sky are more childlike and cheerful"), voice emotion vector The system integrates features such as "cheerful" (e.g., "twinkle twinkle little star") and structured lyrics (e.g., "twinkle twinkle little star") to generate MIDI scores (e.g., "cheerful, childlike"). The generated vocal scores will take into account the user's preferred style to meet the user's personalized creative needs.

[0096] Step 4: Vocal Audio Generation [Input]: Structured lyrics, vocal scores, and semantically irrelevant user acoustic features.

[0097] [Output]: Human voice audio.

[0098] 1. The vocal audio generation model is based on the lyrics generated in step two, the vocal score generated in step three, and the semantically unrelated user acoustic features extracted in step one. Generates user-generated vocal audio (i.e., human voice audio).

[0099] 2. Human voice audio is generated in a diffusion manner. To ensure pronunciation accuracy, a word-to-phoneme (G2P) module is introduced to parse the lyrics and map them into phoneme information. Dynamic time warping (DTW) alignment is used to match the decomposed phonemes with the notes recorded in the electronic score. The vocal audio generation model will predict the singing pronunciation of each phoneme in turn.

[0100] Step 5: Generating Arrangement and Music [Input]: Speech semantic vector (For example, information such as "childlike and cheerful"), human voice audio, and voice emotion vectors. (For example, it contains children's voices and cheerful messages in the sound).

[0101] [Output]: Arrangement and music.

[0102] 1. Based on the speech semantic vectors extracted in step one The human voice audio generated in step four and the voice emotion vector extracted in step one. This generates the corresponding arrangement and music. The semantic information can contain descriptive information about musical preferences, such as "relatively childlike and cheerful," meaning that the target arrangement information can be determined based on the speech and semantics.

[0103] 2. For the human voice audio generated in step four, first encode it using an audio encoder to obtain... At the same time, the BERT model is used to analyze the results obtained in step one. Perform secondary encoding, and... Secondary encoding is performed, integrating the above three features as guiding conditions for diffusion, to generate the arrangement and music.

[0104] 3. In addition, user listening preference data can be used as input conditions for the model to constrain the generated melody to conform to the user's listening habits. Figure 8 This is a schematic diagram of the music arrangement and composition generation process according to an embodiment of this application, such as... Figure 8 As shown, the collected user preference reference songs are processed by an Audio Encoder to obtain corresponding reference vectors. This is fed into the diffusion acoustic model and compared with the aforementioned features (i.e., speech semantic vectors). (For example, "songs that express a love for the starry sky are more childlike and cheerful"), voice emotion vector (e.g., "cheerful"), human voice audio encoding Feature fusion is performed to generate arrangement and music, and user preference style will be taken into account in the generated arrangement and music information.

[0105] Step Six: Alignment and Output Align and output the vocal audio generated in step four and the arranged and matched audio generated in step five to obtain the final complete song audio with arrangement and matched music that conforms to the user's semantic description and uses the user's timbre characteristics for singing (corresponding to the aforementioned target music).

[0106] This application proposes a novel method for music creation that automatically generates personalized songs based on user voice input. This method not only captures the user's semantic information but also generates corresponding vocal parts based on the user's vocal characteristics, while simultaneously arranging accompaniment that matches the user's input. This technology has broad application prospects, extending beyond music production to other fields such as education, advertising, and gaming.

[0107] To implement the music generation method of this application embodiment, this application embodiment also provides a music generation apparatus. Figure 9 This is a schematic diagram of the composition and structure of the music generation device according to an embodiment of this application, as shown below. Figure 9 As shown, the music generation device includes: Feature extraction unit 91 is used to extract features from the speech information of the target object to obtain the speech semantic features and speech acoustic features of the target object; The first generation unit 92 is used to generate human voice audio based on the speech semantic features and the speech acoustic features; The second generation unit 93 is used to generate music arrangement audio based on the human voice audio; The third generation unit 94 is used to generate target music based on the human voice audio and the arrangement and background music audio.

[0108] In one embodiment, the first generation unit 92 includes a fourth generation unit, a fifth generation unit, and a sixth generation unit; wherein, The fourth generation unit is used to generate lyrics text information based on the speech semantic features; The fifth generation unit is used to generate a human voice score based on the speech semantic features; The sixth generation unit is used to generate the human voice audio based on the lyrics text information, the human voice score, and the speech acoustic features.

[0109] In one embodiment, the fourth generation unit is specifically used for: Emotion recognition is performed on the speech information of the target object to obtain the speech emotion features of the target object; The lyrics text information is generated based on the speech semantic features and the speech emotional features.

[0110] In one embodiment, the fifth generating unit is specifically used for: Emotion recognition is performed on the speech information of the target object to obtain the speech emotion features of the target object; The vocal score is generated based on the speech semantic features, the lyrics text information, and the speech emotional features.

[0111] In another embodiment, the fifth generating unit is further specifically used for: Obtain the preference data of the target object; The vocal score is generated based on the speech semantic features, the lyrics text information, the speech emotional features, and the target object's preference data.

[0112] In one embodiment, the second generation unit 93 includes an emotion recognition unit and a seventh generation unit; wherein, The emotion recognition unit is used to perform emotion recognition on the speech information of the target object to obtain the speech emotion features of the target object; The seventh generation unit is used to generate the music arrangement audio based on the speech semantic features, the human voice audio, and the speech emotional features.

[0113] In one embodiment, the seventh generating unit is specifically used for: Obtain the preference data of the target object; The music arrangement and background music are generated based on the speech semantic features, the human voice audio, the speech emotional features, and the target object's preference data.

[0114] In practical applications, the feature extraction unit 91, the first generation unit 92, the second generation unit 93, and the third generation unit 94 can be implemented by the processor in the music generation device.

[0115] It should be noted that the music generation device provided in the above embodiments is only illustrated by the division of the above-described program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the music generation device and the music generation method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the music generation method embodiments, which will not be repeated here.

[0116] Based on the hardware implementation of the above program modules, and in order to implement the music generation method of this application embodiment, this application embodiment also provides an electronic device. Figure 10 This is a schematic diagram of the hardware composition structure of the electronic device according to an embodiment of this application, such as... Figure 10 As shown, the electronic device 1000 includes: Communication interface 1001 enables information exchange with other devices; The processor 1002 is connected to the communication interface 1001 to enable information interaction with other devices. When running a computer program, it executes the music generation method provided above, and the computer program is stored in the memory 1003.

[0117] It should be noted that the specific processing procedures of the communication interface 1001 and the processor 1002 can be understood by referring to the music generation method described above.

[0118] Of course, in practical applications, the various components in electronic device 1000 are coupled together through bus system 1004. It is understood that bus system 1004 is used to realize the connection and communication between these components. In addition to a data bus, bus system 1004 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in... Figure 10 The general labeled all buses as Bus System 1004.

[0119] The memory 1003 in this embodiment is used to store various types of data to support the operation of the electronic device 1000. Examples of such data include any computer program used to operate on the electronic device 1000.

[0120] The music generation method disclosed in the above embodiments of this application can be applied to the processor 1002, or implemented by the processor 1002. The processor 1002 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above music generation method can be completed by the integrated logic circuit of the hardware in the processor 1002 or by instructions in the form of software. The processor 1002 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 1002 can implement or execute the various music generation methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the music generation method disclosed in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in the memory 1003. The processor 1002 reads the information in the memory 1003 and combines its hardware to complete the steps of the aforementioned music generation method.

[0121] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned music generation method.

[0122] It is understood that the memory 1003 in this embodiment can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 1003 described in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0123] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 1003 storing a computer program. This computer program can be executed by a processor 1002 in an electronic device 1000 to complete the steps of the music generation method described in the aforementioned embodiment. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0124] In an exemplary embodiment, this application also provides a computer program product, including a computer program that can be executed by a processor 1002 in an electronic device 1000 to perform the steps of the music generation method described in the foregoing embodiments of this application.

[0125] It should be noted that terms such as "first," "second," and "third" are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0126] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0127] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating music, characterized in that, The method includes: Feature extraction is performed on the speech information of the target object to obtain the speech semantic features and speech acoustic features of the target object; Based on the speech semantic features and the speech acoustic features, generate human voice audio; Based on the human voice audio, generate the arranged background music audio; The target music is generated based on the human voice audio and the arranged background music audio.

2. The method according to claim 1, characterized in that, The process of generating human voice audio based on the speech semantic features and the speech acoustic features includes: Based on the aforementioned speech and semantic features, lyrics text information and vocal scores are generated; The human voice audio is generated based on the lyrics text information, the vocal score, and the speech acoustic features.

3. The method according to claim 2, characterized in that, The process of generating lyrics text information based on the speech semantic features includes: Emotion recognition is performed on the speech information of the target object to obtain the speech emotion features of the target object; The lyrics text information is generated based on the speech semantic features and the speech emotional features.

4. The method according to claim 2, characterized in that, The process of generating a human voice score based on the speech semantic features includes: Emotion recognition is performed on the speech information of the target object to obtain the speech emotion features of the target object; The vocal score is generated based on the speech semantic features, the lyrics text information, and the speech emotional features.

5. The method according to claim 4, characterized in that, The process of generating the vocal score based on the speech semantic features, the lyrics text information, and the speech emotional features includes: Obtain the preference data of the target object; The vocal score is generated based on the speech semantic features, the lyrics text information, the speech emotional features, and the target object's preference data.

6. The method according to claim 1, characterized in that, The process of generating arranged background music audio based on the human voice audio includes: Emotion recognition is performed on the speech information of the target object to obtain the speech emotion features of the target object; The musical arrangement and background music are generated based on the speech semantic features, the human voice audio, and the speech emotional features.

7. The method according to claim 6, characterized in that, The process of generating the music arrangement audio based on the speech semantic features, the human voice audio, and the speech emotional features includes: Obtain the preference data of the target object; The music arrangement and background music are generated based on the speech semantic features, the human voice audio, the speech emotional features, and the target object's preference data.

8. An electronic device, characterized in that, include: A processor and a memory for storing computer programs capable of running on the processor; When the processor is used to run the computer program, it performs the steps of the method according to any one of claims 1 to 7.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images