Music generation method and device, electronic equipment and storage medium

Abstract

The invention provides a music generation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a text prompt word transmitted by a user, and enabling the text prompt word to be used for determining a music element of to-be-generated music; processing the text cue word by adopting a music score generation model to generate an initial music score; modifying the initial music score according to the modification opinion transmitted by the user until no modification opinion is generated, and obtaining a target music score; and processing the target music score by adopting a music generation model to generate target music. According to the invention, the controllability of the generated music is improved.

Description

Technical Field

[0001] The present application relates to the field of large model technology, and in particular to a music generation method, device, electronic device and storage medium. Background Art

[0002] Video soundtracks sometimes require the generation of new music, but generating music often takes a lot of time and effort. For example, when composing music for film and television dramas, creative teachers may face the problem of lack of inspiration, resulting in a long time for the soundtrack. Or when composing music for short videos and advertisements, professional teachers also need to spend a lot of time and money to create.

[0003] Most existing music generation solutions directly generate music by inputting text prompts, but this has low controllability over the music, leaves little room for creators to modify it, and may not meet the creators' various creative needs. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides a music generation method, device, electronic device and storage medium.

[0005] In a first aspect, the present application provides a music generation method, the method comprising:

[0006] Acquire a text prompt word input by a user, wherein the text prompt word is used to determine a music element of the music to be generated;

[0007] Using a music score generation model to process the text prompt words to generate an initial music score;

[0008] Modifying the initial music score according to the modification suggestions input by the user until no modification suggestions are generated, thereby obtaining a target music score;

[0009] The target music score is processed using a music generation model to generate target music.

[0010] Optionally, the text prompt words include main melody prompt words, accompaniment prompt words and instrumentation prompt words, and the using of the music score generation model to process the text prompt words to generate an initial music score includes:

[0011] The music score generation model is used to analyze the main melody prompt words to generate the main melody, wherein the main melody prompt words are used to determine the basic framework and style of the main melody;

[0012] The accompaniment prompt word is analyzed by using the music score generation model to generate an accompaniment coordinated with the main melody, wherein the accompaniment prompt word is used to generate at least one of a chord and an accompaniment;

[0013] The music score generation model is used to analyze the instrumentation prompt words and configure the instrumentation, wherein the timbre and playing style of each instrument in the instrumentation match the overall music style;

[0014] The main melody, the accompaniment and the instrumentation are integrated together to generate a complete initial music score.

[0015] Optionally, the music score generation model is used to analyze the main melody prompt words, and generating the main melody includes:

[0016] If the main melody prompt word includes at least one of a music style, a music theme, a music speed or a music beat, the music score generation model is used to generate the main melody according to the main melody prompt word.

[0017] Optionally, the music score generation model is used to analyze the main melody prompt words, and generating the main melody includes:

[0018] If the main melody prompt word is a music score template, the music features in the music score template are learned by the music score generation model, wherein the music features include music style, music structure, music harmony and music rhythm;

[0019] After learning is completed, a main melody that conforms to the musical characteristics but is innovative is generated.

[0020] Optionally, the target music score is processed by using a music generation model to generate the target music, including:

[0021] Extracting music score features in the target music score through an encoder to obtain a semantic vector, wherein the semantic vector is used to represent music information in the target music score;

[0022] The semantic vector is analyzed by a vocoder to generate the target music.

[0023] Optionally, extracting music score features in the target music score through an encoder to obtain a semantic vector includes:

[0024] Extracting music score features from the target music score;

[0025] Mapping high-dimensional data in the music score features to discrete vectors in a low-dimensional space by compressing high-dimensional feature vectors to obtain music score vectors, wherein the music score vectors are used to represent key features or patterns of the target music score using a simple structure;

[0026] The high-level features in the music score vector are extracted through a target semantic model to generate a semantic vector with semantic meaning, wherein the target semantic model is used to understand and generate music features with semantic meaning.

[0027] Optionally, analyzing the semantic vector by a vocoder to generate the target music includes:

[0028] Converting high-level semantic features in the semantic vector into low-level acoustic features through an acoustic model, wherein the acoustic features are used to characterize the sound characteristics of music;

[0029] The acoustic features are converted into actual audio signals through a decoder to generate the target music.

[0030] In a second aspect, the present application provides a music generation device, the device comprising:

[0031] An acquisition module, used for acquiring text prompt words input by a user, wherein the text prompt words are used for determining the music elements of the music to be generated;

[0032] A first processing module is used to process the text prompt word using a music score generation model to generate an initial music score;

[0033] A modification module, used to modify the initial music score according to the modification suggestions input by the user until no modification suggestions are generated, thereby obtaining a target music score;

[0034] The second processing module is used to process the target music score using a music generation model to generate target music.

[0035] In a third aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0036] Memory, used to store computer programs;

[0037] The processor is used to implement any of the steps of the music generation method when executing the program stored in the memory.

[0038] In a fourth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the steps of the music generation method is implemented.

[0039] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:

[0040] The method provided in the embodiment of the present application divides the music generation process into two parts: score generation and score conversion into audio. During the score generation process, the user can put forward modification suggestions, and the model is automatically modified according to the modification suggestions, thereby realizing the interaction between the model and the user and giving the user a larger modification space, so that the generated score meets the user's creative needs, thereby improving the controllability of the final generated music. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 A schematic diagram of a hardware environment for a music generation method provided in an embodiment of the present application;

[0044] Figure 2 A flowchart of a music generation method provided in an embodiment of the present application;

[0045] Figure 3 An overall system diagram of a music generation method provided in an embodiment of the present application;

[0046] Figure 4 A schematic diagram of the process of generating a music score provided in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of a process for processing a target music score provided in an embodiment of the present application;

[0048] Figure 6 A schematic diagram of the encoder processing flow provided in an embodiment of the present application;

[0049] Figure 7 A schematic diagram of the process flow of the vocoder processing provided in the embodiment of the present application;

[0050] Figure 8 A schematic diagram of the structure of a music generating device provided in an embodiment of the present application;

[0051] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0053] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application and have no specific meaning. Therefore, "module" and "component" can be used interchangeably.

[0054] In order to solve the problems mentioned in the background technology, according to one aspect of the embodiments of the present application, an embodiment of a music generation method is provided.

[0055] Optionally, in the embodiment of the present application, the above-mentioned music generation method can be applied to Figure 1 In the hardware environment composed of the terminal 101 and the server 103 shown in FIG. Figure 1 As shown, the server 103 is connected to the terminal 101 via a network, and can be used to provide services for the terminal or a client installed on the terminal. A database 105 can be set on the server or independently of the server to provide data storage services for the server 103. The above-mentioned network includes but is not limited to: a wide area network, a metropolitan area network or a local area network, and the terminal 101 includes but is not limited to a PC, a mobile phone, a tablet computer, etc.

[0056] A music generation method in an embodiment of the present application can be executed by the server 103 or by the terminal 101.

[0057] The following will be combined with specific implementation methods to provide a detailed description of a music generation method provided in an embodiment of the present application, taking application to a server as an example. Figure 2 As shown, the specific steps are as follows:

[0058] Step 201: obtaining text prompt words input by the user, wherein the text prompt words are used to determine the music elements of the music to be generated;

[0059] Step 202: using a music score generation model to process the text prompt words to generate an initial music score;

[0060] Step 203: modify the initial music score according to the modification suggestions input by the user until no modification suggestions are generated, thereby obtaining the target music score;

[0061] Step 204: Use the music generation model to process the target music score to generate the target music.

[0062] The user enters a text prompt word at the terminal, and the text prompt word is used to determine the musical elements of the music to be generated, for example, the text prompt word is "I need a cheerful main melody, with warm guitar chords, the style is pop music, the speed is medium (120BPM), and the beat is 4 / 4", or "I need a tranquil folk main melody, with simple guitar chords, the style is folk music, the speed is slow (80BPM), and the beat is 4 / 4".

[0063] The terminal sends the text prompt words to the big model, which consists of two parts: the score generation model and the music generation model. The score generation model generates an initial score based on the text prompt words, and then displays the initial score to the user. If the user is not satisfied with the initial score, he or she can put forward modification suggestions. These suggestions can help users express their needs more clearly to further improve the score so as to generate the final target score.

[0064] For example, if the modification suggestion is "I want the main melody to be softer and a little sad", the system will lower the pitch of the melody, add some sustain and diminuendo effects, and use more minor chords. Or if the modification suggestion is "I want to add some string and percussion elements", the system will add string parts to the accompaniment and add some percussion rhythms, such as snare drums and tambourines.

[0065] Ways for users to input modification opinions include, but are not limited to: directly inputting modification opinions in a text input box, inputting modification opinions by filling out a form, inputting modification opinions by selecting an option in a menu, writing modification opinions into a file and then uploading the file, or inputting modification opinions by voice.

[0066] The system makes a modification each time it detects a round of modification opinions. After multiple rounds of iterations, when no modification opinions are detected, the target score is finally generated. The ways in which no modification opinions are detected include: no modification opinions are detected within a certain period of time, or the user selects the option of "confirming that no modification is required".

[0067] This application proposes an innovative music generation method, such as Figure 3As shown in the figure, the method divides the entire music creation process into two main stages: score generation and score conversion to audio. This phased approach not only optimizes the process of music creation, but also greatly enhances the user's participation and control over the final work. In the score generation stage, the system first uses advanced algorithms to automatically generate preliminary scores based on the user's basic requirements (such as musical style, mood, rhythm, etc.). In this step, the system can quickly provide a variety of possible musical ideas as a starting point for creation. Next, users are given great freedom to review and modify these automatically generated scores. They can directly adjust elements such as notes, rhythm, harmony, etc. through an intuitive user interface, and even add or delete certain parts to better express their creativity and emotions. In addition, the system also provides a wealth of editing tools and options to help users carve every detail more finely to ensure that the score can accurately reflect their creative intentions. After completing the customization of the score, it enters the second stage: score conversion to audio. In this stage, highly developed audio synthesis technology is used to convert the carefully adjusted score into a high-quality audio file.

[0068] This application divides music generation into two parts: score generation and score conversion to audio. During the score generation process, users can propose modification suggestions, and the model automatically modifies according to the modification suggestions, realizing the interaction between the model and the user, giving the user a large modification space, so that the generated score meets the user's creative needs, thereby improving the controllability of the final generated music. In addition, this application uses a model to speed up music generation and save production costs.

[0069] As an optional implementation, the text prompt words include main melody prompt words, accompaniment prompt words and instrument prompt words. The text prompt words are processed by the music score generation model, and the generation of the initial music score includes the following four steps:

[0070] The music score generation model is used to analyze the main melody prompt words to generate the main melody, wherein the main melody prompt words are used to determine the basic framework and style of the main melody;

[0071] The accompaniment prompt word is analyzed by using a music score generation model to generate an accompaniment coordinated with the main melody, wherein the accompaniment prompt word is used to generate at least one of a chord and an accompaniment;

[0072] The music score generation model is used to analyze the instrumentation prompt words and configure the instrumentation, wherein the timbre and playing style of each instrument in the instrumentation match the overall music style;

[0073] Integrate the main melody, accompaniment and orchestration together to generate a complete initial score.

[0074] The score generation module uses the ChatGPT large language model to generate scores. It mainly includes three parts: the main melody composer, the accompaniment composer, and the instrumentation composer. The generation process of each part is mutually collaborative, and finally a complete score is generated. The following is a detailed workflow:

[0075] 1. Theme composer

[0076] User input prompt words: Users can provide a variety of prompt words according to their needs, including style, theme, speed, beat and music template, etc.

[0077] Style: Such as classical, jazz, pop, etc.

[0078] Theme: such as love, nature, festivals, etc.

[0079] Speed: such as adagio, allegro, etc.

[0080] Beat: such as 4 / 4, 3 / 4, etc.

[0081] Music score template: If there is a specific music score template, it can be provided to the large model for learning.

[0082] Generate the main melody: The ChatGPT large language model generates the main melody based on the style, theme, speed or beat provided by the user; if a music score template is provided, the model will learn the style and structure of the template; if both a music score template and a style, theme, etc. are provided, the model will combine this information to create.

[0083] 2. Accompaniment Composer

[0084] User-selected accompaniment type: The user can choose to generate chords or accompaniment, or both.

[0085] Generate chords and accompaniment: The ChatGPT large language model generates chords and accompaniment based on the main melody and the user's choice. The model ensures that the chords and accompaniment are consistent with the main melody to enhance the harmony of the music.

[0086] 3. Instrumental composer

[0087] User-specified instruments: Users can choose their favorite instruments, such as piano, violin, flute, etc.

[0088] Configuring instruments: The ChatGPT large language model configures harmonious instruments for music based on the main melody and accompaniment. The model ensures that the timbre and playing style of each instrument match the overall musical style.

[0089] 4. Music combination

[0090] The generated main melody, chords, accompaniment and instrumentation scores will be integrated together to form a complete score. Each part is coordinated in rhythm, harmony and timbre to form a harmonious musical work.

[0091] Figure 4 This is a flowchart of score generation. It can be seen that after the text prompt is input, the initial score is generated by the main melody composer, accompaniment composer and orchestration composer in turn. The user will evaluate the generated score and give suggestions for improvement. The system analyzes which part of the modification suggestions the user has made, and then transmits the modification suggestions to the corresponding composer. The system automatically makes modifications until the user's requirements are met.

[0092] Among them, the main melody composer generates the main melody in three ways, including the following:

[0093] The first method: if the main melody prompt word includes at least one of the music style, music theme, music speed or music beat, the music score generation model generates the main melody according to the main melody prompt word.

[0094] The music score generation model will innovate and create a unique main melody based on the prompt words. The generated main melody will meet the style, theme, speed and beat requirements provided by the user. For example, if the prompt words entered by the user are style: classical, theme: love, speed: slow, beat: 4 / 4, then a classical style slow main melody will be generated.

[0095] The second method: If the main melody prompt is a music score template, the music features in the music score template are learned through the music score generation model, where the music features include music style, music structure, music harmony and music rhythm; after the learning is completed, a main melody that meets the music features but is innovative is generated.

[0096] The user provides a specific music score template, which can be an existing music fragment. The music score generation model will analyze the style, structure, harmony, rhythm and other features of the template, and then extract the key features of the template, such as melody lines, harmonic progressions, rhythmic patterns, etc. The music score generation model generates a new main melody based on the extracted features. The generated main melody will try to maintain a style and structure similar to the template, but there will be certain innovations and changes to avoid complete copying.

[0097] For example, the music score generation model will learn the musical style of the music score template, such as the solemnity of classical music and the improvisation of jazz music; learn the form structure of the music score template, such as ABA form and sonata form; learn the harmonic progression of the music score template, such as common chord progressions; learn the rhythmic pattern of the music score template, such as the distribution of strong and weak beats.

[0098] The main melody generated by the music score generation model will maintain a style similar to the template, but will have certain innovations; the generated main melody will follow a similar form structure, but will have changes; the generated main melody will be coordinated with the harmony of the template, but will have new chord progressions; the generated main melody will maintain a rhythmic pattern similar to the template, but will have new rhythmic changes.

[0099] The third method: The main melody prompts include both the music score template and the music style, music theme, etc. Then, the generated main melody, while conforming to the music score template, is further guided by the style, theme, etc. in the generation process. These prompts can help the model make more specific adjustments when generating the main melody.

[0100] Whether a music score template is provided or not, the model will generate a main melody based on the information provided by the user. When a music score template is provided, the model will learn the style and structure of the template and generate a main melody that is similar to the template but innovative; when a music score template is not provided, the model will create innovatively based on other information provided by the user (such as style, theme, speed, beat, etc.) to generate a unique main melody.

[0101] Among them, there are three ways for the accompaniment composer to generate accompaniment, including the following:

[0102] The first method is to generate chords to support the harmony of the main melody. The selection of chords will take into account the mode, scale and melody direction of the main melody, and the generated chords will be presented in the form of standard chord notation, such as Cmaj7, Am7, F#dim7, etc.

[0103] The second method is to generate an accompaniment to provide background music and rhythm support. The rhythm pattern can be simple quarter notes, eighth notes, or complex syncopated rhythms. The accompaniment will provide harmonic support to enhance the harmonic effect of the main melody. The harmonic support can be simple chord accompaniment or complex harmonic texture.

[0104] Third way: Generate chords and accompaniment simultaneously for full harmonic and rhythmic support.

[0105] As an optional implementation, a music generation model is used to process the target music score, and generating the target music includes: extracting music score features in the target music score through an encoder to obtain a semantic vector, wherein the semantic vector is used to represent the music information in the target music score; and analyzing the semantic vector through a vocoder to generate the target music.

[0106] Figure 5The flowchart for processing the target score shows that the Token encoder receives digital score data as input, which may include notes, rhythm, pitch, duration, dynamic markers, etc. Through the processing of multi-layer neural networks, the encoder extracts information that can represent the core features of the target score, and then converts these specific music symbols into a higher-level abstract representation, namely a semantic vector. The semantic vector is a multi-dimensional mathematical expression that captures the musical information in the score, such as melody lines, harmonic progressions, rhythmic patterns, etc. The semantic vector is a bridge between the score and the final audio output. It not only contains the basic structural information of the score, but also contains deeper content such as the emotional color and style characteristics of the music. This high-level abstraction enables the model to better understand and simulate the way humans perceive music, thereby generating more natural and expressive music.

[0107] Transformer vocoders usually contain complex generative networks that can gradually construct the various components of music, such as timbre, pitch, volume, duration, etc., based on the input semantic vectors. Once the vocoder completes the analysis of the semantic vectors, it will begin to generate the target music. This process may involve multiple iterations and optimization steps to ensure the quality and coherence of the music. The generated music will undergo post-processing, such as mixing and mastering, to further improve its sound quality and listening experience. In the end, users will get a complete, high-quality piece of music that retains the core features of the original score while incorporating the model's unique creativity and interpretation.

[0108] Among them, extracting the music score features in the target music score through the encoder to obtain the semantic vector includes: extracting the music score features in the target music score; mapping the high-dimensional data in the music score features to discrete vectors in the low-dimensional space by compressing the high-dimensional feature vector to obtain the music score vector, wherein the music score vector is used to represent the key features or patterns of the target music score with a simple structure; extracting high-level features in the music score vector through the target semantic model to generate a semantic vector with semantic meaning, wherein the target semantic model is used to understand and generate music features with semantic meaning.

[0109] Figure 6This is a flowchart of the encoder processing. It can be seen that the encoder extracts the music features of the target music score, such as melody, harmony, rhythm, etc., and then uses RVQ (Residual Vector Quantization) technology to compress and quantize the high-dimensional feature vector into a low-dimensional music score token, outputs the music score token (music score vector), and then uses the Semantic Model (target semantic model) to extract higher-level semantic features from the music score token to generate a sementic token (semantic vector). The sementic token can represent the style, emotion, structure, etc. of the music.

[0110] Optionally, RVQ is a technology for compressing high-dimensional feature vectors by mapping high-dimensional features into a low-dimensional discrete codebook, thereby reducing data storage and transmission costs.

[0111] Optionally, the score token is a discrete vector obtained by compressing and quantizing the score features through RVQ technology. Each token represents a part of the score features, such as melody, harmony, rhythm, etc. The score token can efficiently represent the features of the score, reducing the complexity of the data and storage requirements.

[0112] Optionally, a Semantic Model is a deep learning model for understanding and generating semantically meaningful music features, which can capture high-level features of music, such as style, emotion, structure, etc. The semantic tokens generated by the Semantic Model can be used to control the generated music to ensure that the generated music meets the needs and expectations of users.

[0113] Optionally, Sementic tokens are vectors with semantic meaning generated by Semantic Model, which can capture high-level features of music, such as style, emotion, structure, etc.

[0114] Among them, the semantic vector is analyzed by a vocoder to generate the target music, including: converting the high-level semantic features in the semantic vector into low-level acoustic features through an acoustic model, wherein the acoustic features are used to characterize the sound characteristics of the music; and converting the acoustic features into actual audio signals through a decoder to generate the target music.

[0115] Figure 7 The following is a flowchart of the vocoder processing. It can be seen that the process from semantic features to final music generation involves two key steps: generating acoustic representation through the Acoustic Model, and generating the final music through the decoder. The following is a detailed explanation of each step:

[0116] Acoustic Model is a deep learning model that converts high-level semantic features (such as SementicToken) into low-level acoustic features. These acoustic features include pitch, timbre, rhythm, etc., which can be directly used to synthesize music.

[0117] The Acoustic Model has two functions. One is feature conversion: converting high-level semantic features in Sementic Token into specific acoustic features, such as spectrum, Mel-frequency cepstral coefficients (MFCC), fundamental frequency, etc. The other is acoustic representation: the generated acoustic features can represent the sound characteristics of music and provide a basis for subsequent music synthesis. Among them, the acoustic representation is a feature vector generated by the Acoustic Model that can represent the sound characteristics of music. These feature vectors include spectrum, MFCC, fundamental frequency, etc. Acoustic representation can accurately describe the sound characteristics of music, such as pitch, timbre, rhythm, etc. Acoustic representation is the basis of music synthesis and is used to generate the final audio signal.

[0118] A decoder is a model that converts acoustic features into audio signals. The decoder can be a neural network-based model, such as WaveNet, Tacotron, etc. The decoder has two functions: one is audio synthesis: converting acoustic representations into actual audio signals to generate the final music; the other is high-quality output: the decoder can generate high-quality audio to ensure that the generated music sounds natural and realistic.

[0119] This application generates acoustic representations from Sementic Tokens through the Acoustic Model, and then generates the final music through the decoder, which can achieve efficient conversion from high-level semantic features to specific audio. The Acoustic Model is responsible for converting semantic features into acoustic features, and the decoder is responsible for converting acoustic features into actual audio signals, ensuring that the generated music is of high quality and meets user needs.

[0120] Based on the same technical concept, the embodiment of the present application also provides a process step of music generation, including the following steps:

[0121] Step 1: The user inputs a text prompt into the system. The system includes a music score generation model and a music generation module. The music score generation model includes a main melody composer, an accompaniment composer, and an instrumentation composer.

[0122] Step 2: The main melody composer generates the main melody according to the main melody prompt words; the accompaniment composer generates the accompaniment according to the accompaniment prompt words; the orchestration composer configures the orchestration according to the orchestration prompt words.

[0123] Step 3: The user proposes modification suggestions, and the composer modifies each part of the score generation model according to the modification suggestions until no modification suggestions are generated, and the target score is generated.

[0124] Step 4: The target music score is passed into the encoder to extract the music score features and convert it into a low-dimensional music score token using RVQ. The Semantic Model captures the high-level features of the music score token and generates a semantic token.

[0125] Step 5: The Acoustic Model converts the sementic token into low-level acoustic features, and the decoder converts the acoustic features into audio signals and outputs music.

[0126] In this application, it can be divided into three modules: 1. ChatGPT score generation module: generate music scores that conform to musical aesthetics from text prompts; 2. Music score MIDI conversion module: generate MIDI (Musical Instrument Digital Interface) files from music scores; 3. MIDI music generation module: use deep learning models to generate music from MIDI files.

[0127] Alternatively, a MIDI file is a file format for storing music information, and the generation of a MIDI file is a step between the generation of a score and the conversion of the score into audio. MIDI files are usually very small because they only contain instructions, not actual audio data; MIDI files can be easily edited, such as changing notes, adjusting tempo, changing timbre, etc.; MIDI files can be used on different hardware and software platforms with good compatibility; MIDI files can be used with various synthesizers and sound sources to generate music of different styles.

[0128] In this application, the score generation module and the music generation mode are both large language models, and the functions of the large language model include the following:

[0129] 1. Feature extraction and understanding

[0130] Feature extraction: The large language model can deeply understand and extract various features in the music score, such as melody, harmony, rhythm, timbre, etc. These features are the basis for generating high-quality music scores.

[0131] Contextual understanding: The large language model can understand the contextual information of the score, such as style, theme, emotion, etc., to ensure that the generated score remains consistent and coherent overall.

[0132] 2. Generate high-quality main melody

[0133] Main melody generation: The large language model can generate high-quality main melody based on the prompt words provided by the user (such as style, theme, speed, beat, etc.). These prompt words can be specific musical elements or abstract emotional descriptions.

[0134] Innovation and diversity: The large language model is able to innovate when generating the main melody, generating diverse melodic lines and avoiding repetition and monotony.

[0135] 3. Generate chords and accompaniment

[0136] Chord generation: The large language model can generate appropriate chords based on the main melody, ensuring that the chords are consistent with the main melody and enhancing the harmonic effect of the music.

[0137] Accompaniment generation: The large language model can generate rich accompaniment, including chord accompaniment, rhythm accompaniment, etc., to enhance the layering and richness of the music.

[0138] 4. Generate instrumentation

[0139] Instrumentation generation: The large language model can configure harmonious instruments for music based on the main melody and accompaniment. Users can specify their favorite instruments, and the large language model will ensure that the timbre and playing style of each instrument match the overall musical style.

[0140] 5. Generate high-quality music

[0141] Acoustic representation: The large language model can convert semantic tokens into specific acoustic representations, such as spectrum, MFCC, fundamental frequency, etc. These acoustic representations can accurately describe the sound characteristics of music.

[0142] Audio synthesis: Through the Acoustic Model and decoder, the acoustic representation is converted into an actual audio signal to generate high-quality music.

[0143] 6. Generate music that meets musical aesthetics

[0144] Emotional expression: The large language model can understand and generate music with specific emotions, such as happiness, sadness, passion, etc. The generated music can better express the emotional needs of users.

[0145] Style consistency: Large language models can generate music that conforms to a specific style, such as classical, jazz, pop, etc. The generated music remains consistent in style and conforms to the user's aesthetic preferences.

[0146] 7. Generate diverse music

[0147] Innovation and diversity: The large language model can be innovative when generating music and generate diverse musical works. The generated music is not only innovative in melody and harmony, but also varied in rhythm and timbre to avoid repetition and monotony.

[0148] By introducing a large language model, the score generation module and the music generation module can better understand and generate music, ensuring that the generated music conforms to musical aesthetics in terms of melody, harmony, rhythm, timbre, etc., and meets the diverse needs of users.

[0149] Based on the same technical concept, the present application also provides a music generating device, such as Figure 8 As shown, the device comprises:

[0150] The acquisition module 801 is used to acquire the text prompt word input by the user, wherein the text prompt word is used to determine the music element of the music to be generated;

[0151] The first processing module 802 is used to process the text prompt words using a music score generation model to generate an initial music score;

[0152] The modification module 803 is used to modify the initial music score according to the modification suggestions input by the user until no modification suggestions are generated, thereby obtaining the target music score;

[0153] The second processing module 804 is used to process the target music score using the music generation model to generate the target music.

[0154] Optionally, the text prompt words include main melody prompt words, accompaniment prompt words and instrumentation prompt words, and the first processing module 802 is used to:

[0155] The music score generation model is used to analyze the main melody prompt words to generate the main melody, wherein the main melody prompt words are used to determine the basic framework and style of the main melody;

[0156] The accompaniment prompt word is analyzed by using a music score generation model to generate an accompaniment coordinated with the main melody, wherein the accompaniment prompt word is used to generate at least one of a chord and an accompaniment;

[0157] The music score generation model is used to analyze the instrumentation prompt words and configure the instrumentation, wherein the timbre and playing style of each instrument in the instrumentation match the overall music style;

[0158] Integrate the main melody, accompaniment and orchestration together to generate a complete initial score.

[0159] Optionally, the first processing module 802 is specifically configured to:

[0160] If the main melody prompt word includes at least one of a music style, a music theme, a music speed or a music beat, a music score generation model is used to generate the main melody according to the main melody prompt word.

[0161] Optionally, the first processing module 802 is specifically configured to:

[0162] If the main melody prompt word is a music score template, the music features in the music score template are learned through the music score generation model, wherein the music features include music style, music structure, music harmony and music rhythm;

[0163] After learning is completed, a main melody that conforms to the musical characteristics but is innovative is generated.

[0164] Optionally, the second processing module 804 is used to:

[0165] Extracting music score features in the target music score through an encoder to obtain a semantic vector, wherein the semantic vector is used to represent music information in the target music score;

[0166] The semantic vector is analyzed through a vocoder to generate the target music.

[0167] Optionally, the second processing module 804 is specifically configured to:

[0168] Extracting music score features from the target music score;

[0169] Mapping high-dimensional data in the music score features to discrete vectors in a low-dimensional space by compressing high-dimensional feature vectors to obtain music score vectors, wherein the music score vectors are used to represent key features or patterns of the target music score using a simple structure;

[0170] The target semantic model is used to extract high-level features from the score vector and generate a semantic vector with semantic meaning, wherein the target semantic model is used to understand and generate music features with semantic meaning.

[0171] Optionally, the second processing module 804 is specifically configured to:

[0172] The high-level semantic features in the semantic vector are converted into low-level acoustic features through an acoustic model, wherein the acoustic features are used to characterize the sound characteristics of music;

[0173] The decoder converts the acoustic features into actual audio signals to generate the target music.

[0174] Based on the same technical concept, an embodiment of the present invention further provides an electronic device, such as Fig. 9 As shown, it includes a processor 901, a communication interface 902, a memory 903 and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.

[0175] Memory 903, used for storing computer programs;

[0176] The processor 901 is used to implement the above steps when executing the program stored in the memory 903.

[0177] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0178] The communication interface is used for communication between the above electronic device and other devices.

[0179] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0180] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0181] In another embodiment of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, the steps of implementing any of the above methods include the following contents:

[0182] Acquire a text prompt word input by a user, wherein the text prompt word is used to determine a music element of the music to be generated;

[0183] Using a music score generation model to process the text prompt words to generate an initial music score;

[0184] Modifying the initial music score according to the modification suggestions input by the user until no modification suggestions are generated, thereby obtaining a target music score;

[0185] The target music score is processed using a music generation model to generate target music.

[0186] Optionally, the text prompt words include main melody prompt words, accompaniment prompt words and instrumentation prompt words, and the using of the music score generation model to process the text prompt words to generate an initial music score includes:

[0187] The music score generation model is used to analyze the main melody prompt words to generate the main melody, wherein the main melody prompt words are used to determine the basic framework and style of the main melody;

[0188] The accompaniment prompt word is analyzed by using the music score generation model to generate an accompaniment coordinated with the main melody, wherein the accompaniment prompt word is used to generate at least one of a chord and an accompaniment;

[0189] The music score generation model is used to analyze the instrumentation prompt words and configure the instrumentation, wherein the timbre and playing style of each instrument in the instrumentation match the overall music style;

[0190] The main melody, the accompaniment and the instrumentation are integrated together to generate a complete initial music score.

[0191] Optionally, the music score generation model is used to analyze the main melody prompt words, and generating the main melody includes:

[0192] If the main melody prompt word includes at least one of a music style, a music theme, a music speed or a music beat, the music score generation model is used to generate the main melody according to the main melody prompt word.

[0193] Optionally, the music score generation model is used to analyze the main melody prompt words, and generating the main melody includes:

[0194] If the main melody prompt word is a music score template, the music features in the music score template are learned by the music score generation model, wherein the music features include music style, music structure, music harmony and music rhythm;

[0195] After learning is completed, a main melody that conforms to the musical characteristics but is innovative is generated.

[0196] Optionally, the target music score is processed by using a music generation model to generate the target music, including:

[0197] Extracting music score features in the target music score through an encoder to obtain a semantic vector, wherein the semantic vector is used to represent music information in the target music score;

[0198] The semantic vector is analyzed by a vocoder to generate the target music.

[0199] Optionally, extracting music score features in the target music score through an encoder to obtain a semantic vector includes:

[0200] Extracting music score features from the target music score;

[0201] Mapping high-dimensional data in the music score features to discrete vectors in a low-dimensional space by compressing high-dimensional feature vectors to obtain music score vectors, wherein the music score vectors are used to represent key features or patterns of the target music score using a simple structure;

[0202] The high-level features in the music score vector are extracted through a target semantic model to generate a semantic vector with semantic meaning, wherein the target semantic model is used to understand and generate music features with semantic meaning.

[0203] Optionally, analyzing the semantic vector by a vocoder to generate the target music includes:

[0204] Converting high-level semantic features in the semantic vector into low-level acoustic features through an acoustic model, wherein the acoustic features are used to characterize the sound characteristics of music;

[0205] The acoustic features are converted into actual audio signals through a decoder to generate the target music.

[0206] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes any one of the methods in the above embodiments.

[0207] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0208] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0209] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A music generation method, characterized in that: The method comprises: Acquire a text prompt word input by a user, wherein the text prompt word is used to determine a music element of the music to be generated; Using a music score generation model to process the text prompt words to generate an initial music score; Modifying the initial music score according to the modification suggestions input by the user until no modification suggestions are generated, thereby obtaining a target music score; The target music score is processed using a music generation model to generate target music.

2. The method according to claim 1, characterized in that The text prompt words include main melody prompt words, accompaniment prompt words and instrument prompt words. The music score generation model is used to process the text prompt words to generate an initial music score, which includes: The music score generation model is used to analyze the main melody prompt words to generate the main melody, wherein the main melody prompt words are used to determine the basic framework and style of the main melody; The accompaniment prompt word is analyzed by using the music score generation model to generate an accompaniment coordinated with the main melody, wherein the accompaniment prompt word is used to generate at least one of a chord and an accompaniment; The music score generation model is used to analyze the instrumentation prompt words and configure the instrumentation, wherein the timbre and playing style of each instrument in the instrumentation match the overall music style; The main melody, the accompaniment and the instrumentation are integrated together to generate a complete initial music score.

3. The method according to claim 2, characterized in that The music score generation model is used to analyze the main melody prompt words, and the main melody is generated including: If the main melody prompt word includes at least one of a music style, a music theme, a music speed or a music beat, the music score generation model is used to generate the main melody according to the main melody prompt word.

4. The method according to claim 2, characterized in that: The music score generation model is used to analyze the main melody prompt words, and the main melody is generated including: If the main melody prompt word is a music score template, the music features in the music score template are learned by the music score generation model, wherein the music features include music style, music structure, music harmony and music rhythm; After learning is completed, a main melody that conforms to the musical characteristics but is innovative is generated.

5. The method according to claim 1, characterized in that The target music score is processed by using a music generation model to generate the target music, including: Extracting music score features in the target music score through an encoder to obtain a semantic vector, wherein the semantic vector is used to represent music information in the target music score; The semantic vector is analyzed by a vocoder to generate the target music.

6. The method according to claim 5, characterized in that The encoder extracts the music score features in the target music score, and obtains the semantic vector including: Extracting music score features from the target music score; Mapping high-dimensional data in the music score features to discrete vectors in a low-dimensional space by compressing high-dimensional feature vectors to obtain music score vectors, wherein the music score vectors are used to represent key features or patterns of the target music score using a simple structure; The high-level features in the music score vector are extracted through a target semantic model to generate a semantic vector with semantic meaning, wherein the target semantic model is used to understand and generate music features with semantic meaning.

7. The method according to claim 5, characterized in that Analyzing the semantic vector by a vocoder to generate the target music includes: Converting high-level semantic features in the semantic vector into low-level acoustic features through an acoustic model, wherein the acoustic features are used to characterize the sound characteristics of music; The acoustic features are converted into actual audio signals through a decoder to generate the target music.

8. A music generating device, characterized in that: The device comprises: An acquisition module, used for acquiring text prompt words input by a user, wherein the text prompt words are used for determining the music elements of the music to be generated; A first processing module is used to process the text prompt word using a music score generation model to generate an initial music score; A modification module, used to modify the initial music score according to the modification suggestions input by the user until no modification suggestions are generated, thereby obtaining a target music score; The second processing module is used to process the target music score using a music generation model to generate target music.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 7 are implemented.

Images