Training method of music generation model and music generation method and device

Abstract

The embodiment of the invention provides a training method of a music generation model and a music generation method and device, and relates to the technical field of artificial intelligence, and the training method of the music generation model comprises the steps: obtaining sample data; the sample data comprises sample note data, sample image data and label information; the label information is used for representing style types corresponding to the sample note data and the sample image data; inputting the sample data into a neural network model for music generation to obtain predicted note data of the sample data; and determining a model loss function of the neural network model according to the predicted note data and the sample data, and performing iterative training on the neural network model based on the model loss function to obtain a music generation model. The music generation model depends on the note data and the image data with the specific style type at the same time in the training stage, so that when the trained music generation model generates music, high-quality music matched with the style type of the given image can be generated.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a training method for a music generation model, a music generation method, and an apparatus. Background Technology

[0002] Currently, artificial intelligence has made significant progress in multimodal content generation. However, in the multimedia field, creating art and music works comparable to those of human composers using AI models remains a complex challenge. Among the various methods used for music generation, such as Markov models, genetic algorithms, generative grammars, and neural networks, neural network models stand out for their flexibility. Neural network models can be trained to capture complex patterns in large music datasets, thus adapting to the complexity of music composition. While many neural network models can effectively analyze musical patterns from sources such as sheet music or audio files and generate impressive compositions, these models rely solely on musical features during both the training and generation phases, failing to utilize visual information to guide the creative process. Therefore, there is still enormous room for development in the field of music generation using AI models. Summary of the Invention

[0003] This application provides a method for training a music generation model, a music generation method, and an apparatus to alleviate or solve one or more technical problems existing in the prior art.

[0004] In a first aspect, embodiments of this application provide a method for training a music generation model, comprising:

[0005] Acquire sample data; the sample data includes: sample note data, sample image data, and label information; the label information is used to indicate the style type corresponding to the sample note data and the sample image data.

[0006] The sample data is input into the neural network model to be trained to generate music, and the predicted note data of the sample data is obtained.

[0007] Based on the predicted note data and the sample data, the model loss function of the neural network model is determined, and the neural network model is iteratively trained based on the model loss function to obtain a music generation model.

[0008] Secondly, embodiments of this application provide a music generation method, including:

[0009] Acquire note data and image data;

[0010] The note data and image data are input into a pre-trained music generation model. The music generation model classifies the image data to obtain the style type corresponding to the image data. Music is generated based on the note data and the style type to obtain target music that conforms to the style type.

[0011] The music generation model is trained according to the training method of the music generation model described in any embodiment of this application.

[0012] Thirdly, embodiments of this application provide a training apparatus for a music generation model, comprising:

[0013] The first acquisition module is used to acquire sample data; the sample data includes: sample note data, sample image data, and label information; the label information is used to indicate the style type corresponding to the sample note data and the sample image data.

[0014] The prediction module is used to input the sample data into the neural network model to be trained to generate music and obtain the predicted note data of the sample data.

[0015] The training module is used to determine the model loss function of the neural network model based on the predicted note data and the sample data, and to iteratively train the neural network model based on the model loss function to obtain a music generation model.

[0016] Fourthly, embodiments of this application provide a music generation apparatus, including:

[0017] The second acquisition module is used to acquire note data and image data;

[0018] The generation module is used to input the note data and image data into a pre-trained music generation model, classify the image data through the music generation model to obtain the style type corresponding to the image data, and generate music based on the note data and the style type to obtain target music that conforms to the style type.

[0019] The music generation model is trained according to the training method of the music generation model described in any embodiment of this application.

[0020] Fifthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.

[0021] Sixthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.

[0022] In a seventh aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the embodiments of this application.

[0023] According to the technical solution of this application embodiment, sample data is acquired, including sample note data, sample image data, and label information. The label information is used to represent the style type corresponding to the sample note data and sample image data. The sample data is input into a neural network model to be trained for music generation, resulting in predicted note data. The model loss function of the neural network model is determined based on the predicted note data and the sample data, and the neural network model is iteratively trained based on the model loss function to obtain a music generation model. It is evident that the music generation model relies not only on note data during the training phase but also on image data with specific style types. This allows note features and image styles to simultaneously guide the model's learning objective, enabling the trained music generation model to generate high-quality music that matches the style type of a given image. This has high application value for customizing high-quality music for specific visual scenes.

[0024] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0025] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0026] Figure 1 A flowchart illustrating the training method of the music generation model provided in an embodiment of this application is shown;

[0027] Figure 2 A schematic diagram illustrating the principle of the training method for the music generation model provided in this application embodiment is shown;

[0028] Figure 3 The diagram shows the structure of the dual-axis LSTM model provided in the embodiment of this application;

[0029] Figure 4A flowchart of the music generation method provided in an embodiment of this application is shown;

[0030] Figure 5 A block diagram of a training apparatus for a music generation model provided in an embodiment of this application is shown;

[0031] Figure 6 A block diagram of a music generation apparatus provided in an embodiment of this application is shown;

[0032] Figure 7 A block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0033] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0034] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0035] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0036] Figure 1 A flowchart illustrating the training method of the music generation model provided in this application embodiment is shown, such as... Figure 1 As shown, the method may include steps S101, S102 and S103.

[0037] Step S101: Obtain sample data; the sample data includes: sample note data, sample image data, and label information, whereby the label information is used to indicate the style type corresponding to the sample note data and sample image data.

[0038] The sample data may include sample musical notes and sample images of various styles. The types of styles are not limited; for example, they may include at least one style such as Classical, Baroque, or Romantic.

[0039] Optionally, the sample note data and sample image data can each carry label information. In this way, during training, the neural network model can learn the association between images and note data of various styles based on the label information carried by the sample note data and sample image data respectively.

[0040] Optionally, the label information can be carried only in the sample note data; that is, the sample image data does not need to carry label information. In this case, sample note data and sample image data of the same style type can be associated, so that when performing the subsequent step S102, the associated sample note data and sample image data are simultaneously input into the neural network model to be trained, so that the neural network model can learn the association between image and note data of each style type based on the association.

[0041] Optionally, when acquiring sample note data, multiple datasets from different musical composers can be selected. Each composer has their unique artistic style, thus including sample note data of various style types in the training sample data. The label information representing the style type can be represented in vector form. During model training, multiple style types can be merged by modifying the element values ​​of the style type vector, thereby combining the desired style type. Subsequently, the style type vector is normalized so that the sum of the vector element values ​​is 1.

[0042] For example, the style type vector for composer 1 is represented as [1,0,0…], indicating that the neural network model only considers the style type of the first composer. The style type vector for composer 2 is represented as [0,1,0…], indicating that the neural network model only considers the style type of the second composer. The combined style type vector for composer 1 and composer 2 is represented as [0.5,0.5,0…], indicating the average fusion of the style types of the first and second composers, with each style type component being 0.5.

[0043] Of course, the fusion of style types is not limited to two. For example, if composer 1, composer 2, composer 3, and composer 4 all belong to the same style type, then the style types of composer 1, composer 2, composer 3, and composer 4 can be merged. The merged style type vector can be represented as [0.25, 0.25, 0.25, 0, 0...]. This vector indicates that the style types of the four composers are considered to an equal degree, with the component of the style type of each of the four composers being 0.25, while the component of other style types is 0.

[0044] Optionally, when acquiring sample image data, image data of various styles can be collected. The style type of the sample image data corresponds to the style type of the sample musical note data, and each style type corresponds to multiple sample musical note data and multiple sample image data. For example, the sample data includes multiple sample musical note data and multiple sample image data for the classical style, multiple sample musical note data and multiple sample image data for the Baroque style, multiple sample musical note data and multiple sample image data for the Romantic style, and so on.

[0045] Step S102: Input the sample data into the neural network model to be trained to generate music and obtain the predicted note data of the sample data.

[0046] The neural network model comprises two parts: an image classification module and a music generation module. In step S102, sample data can be input into the neural network model to be trained. The image classification module classifies the style of the sample image data to obtain the predicted style type, which is then input into the music generation module. The music generation module then predicts the sample note data based on the sample note data and the predicted style type, resulting in predicted note data.

[0047] Optionally, the predicted note data includes note data for multiple time steps. When predicting the sample note data based on the sample note data and the predicted style type to obtain the predicted note data, the following steps can be performed: extracting note features from the sample note data; fusing the note features, the predicted style type, and the note data from the previous time step to obtain fused data; and predicting the note data for the current time step of the sample note data based on the fused data. After predicting the notes for each time step, the combination of notes from all time steps constitutes the predicted note data.

[0048] Step S103: Based on the predicted note data and sample data, determine the model loss function of the neural network model, and iteratively train the neural network model based on the model loss function to obtain the music generation model.

[0049] In this embodiment, the trained music generation model can associate image data with different style types, generate style type information for embedding, and guide the generation of music of a specific style.

[0050] According to the technical solution of this application embodiment, sample data is acquired, including sample note data, sample image data, and label information. The label information is used to represent the style type corresponding to the sample note data and sample image data. The sample data is input into a neural network model to be trained for music generation, resulting in predicted note data. The model loss function of the neural network model is determined based on the predicted note data and the sample data, and the neural network model is iteratively trained based on the model loss function to obtain a music generation model. It is evident that the music generation model relies not only on note data during the training phase but also on image data with specific style types. This allows note features and image styles to simultaneously guide the model's learning objective, enabling the trained music generation model to generate high-quality music that matches the style type of a given image. This has high application value for customizing high-quality music for specific visual scenes.

[0051] Figure 2 A schematic diagram illustrating the principle of the training method for the music generation model provided in this application embodiment is shown. Figure 2 As shown, the neural network model to be trained consists of two parts: an image classification module and a music generation module. The image classification module classifies the style of the sample image data to obtain the predicted style type, and then inputs the predicted style type into the music generation module. The music generation module predicts the predicted note data based on the sample note data and the predicted style type.

[0052] The music generation module employs the network structure of the DeepJ music generation model. DeepJ is a deep learning model capable of creating polyphonic music while capturing a single style or a fusion of multiple styles. DeepJ incorporates mechanisms to ensure that the generated music accurately reflects the intended musical style. The music generation module has a dual-axis architecture, including a time axis and a sequence axis. In the time axis, n(tm) represents the note features at time step t, a(tm) represents the output data at time step t, i.e., the predicted note data, and h(tm) represents the hidden state.

[0053] The image classification module is essentially a convolutional neural network structure with image classification capabilities. When processing image data, it encodes the image data into vectors containing style type information. This vector passes through a linear hidden layer in the convolutional neural network structure, which transforms the vector into style type information and embeds it into the music generation module. Optionally, the image classification module includes a classification model and a fully connected hidden layer (also known as the style type embedding layer for image data). The classification model can be any common classification model, such as AlexNet. After AlexNet classifies the image data, it enables the image to be associated with a specific style. Then, the fully connected hidden layer transforms the classification result of the image data into style type information and embeds it into the music generation module, thereby guiding the music generation module to generate music of the corresponding style type.

[0054] Optionally, the music generation module predicts the probability of each note appearing at each time step during the music generation process by considering all previous time steps and the notes generated in the current time step, thereby achieving prediction of polyphonic music. The music generation process proceeds sequentially from the first time step (t=1) to the last time step (t=T), and sequentially from the lowest note (n=1) to the highest note (n=N), where T represents the expected note length and N represents the total number of possible notes. It is evident that the neural network model considers both the time dimension and the note dimension when determining the probability of note appearance.

[0055] Optionally, the music generation module is implemented using a dual-axis LSTM (Long Short-Term Memory) model architecture. For example... Figure 3 As shown, in this architecture, the model input includes the notes from the previous time step. The Note Octaves layer expands each note into a tensor containing the note itself and its neighboring notes within an octave, thus enriching the spatial context of the data. Each note feature is then processed by an LSTM unit, with notes sharing weights along the time axis. Subsequently, the time axis module combines these octave-expanded notes with the cyclic state from the previous time step to generate higher-level features for each individual note. Figure 3 C, C#, D, D#, E, F, and F# are all exemplary musical note symbols.

[0056] Using shared LSTM weights in the timeline module offers a key advantage: modulation invariance. In music, the relative pitch of harmony and melody is more important than absolute pitch because a piece of music remains essentially unchanged when modulated. This concept is similar to translation invariance in image classification, where the position of an image within a frame does not affect its recognizability. Similarly, modulation invariance ensures that the inherent structure and features of music are preserved regardless of key changes.

[0057] Inspired by convolutional neural networks, the timeline module consists of stacked LSTM units capable of processing time-varying sequential data. These units are connected to the octave interval (12 pitches above and below each note) of each note in a manner similar to convolutional kernels. By sharing weights among notes, the LSTM units enable the timeline module to learn features independent of specific notes, thus improving generalization across different key changes. The timeline module then outputs features for each note.

[0058] The note features generated by the timeline module are input into the noteline module. The LSTM units of the noteline module process each note sequentially, starting from the lowest note and predicting progressively from the lowest to the highest note. Each note prediction references previously predicted lower notes, thus enhancing contextual coherence. The noteline module consists of two layers of LSTM units that process the notes cyclically. Before input, each note feature is combined with the features of previously predicted lower notes. For the lowest note, a zero value is added as a placeholder. Specifically, for the predicted note y... i It will take the feature x of the i-th note i The characteristic x of the previous lower note i-1 The styles are then combined and fed into the note axis module. Within each LSTM layer, connections are established between style embeddings, optionally using fully connected hidden layers with tanh activation functions (i.e., style type embedding layers). This connection method enables the model to generate non-linear representations specific to a particular style, for the i-th note and its corresponding note feature x. i The LSTM output z of this note i This can be expressed as the following formula:

[0059] z i =f(X) i +tanhW l ′Ws)

[0060] Where, x i The input is represented by tanhW, where "+" indicates element-wise addition. f represents the function to execute the LSTM layer on the given input. l' represents a fully connected hidden layer with a tanh activation function in the LSTM layer, W represents a linear hidden layer, and s represents style type information. The LSTM layer processing method represented by the above formula ensures that style type constraints are consistently applied to each note.

[0061] In addition to the main data input, the dual-axis architecture can provide each LSTM unit in the timeline module with additional contextual information representing the current beat's position within a measure. This contextual information is encoded in binary format with respect to the relative position of the current time step with respect to the 4 / 4 beat, helping the model capture rhythmic structure.

[0062] It is understandable that the neural network model to be trained introduces an extended framework (i.e., an image classification module) on the basis of DeepJ to support image input, thereby providing style type embeddings to the music generation module through image data.

[0063] In this embodiment, the note data includes note activation data, note ensemble data, and note velocity. Note activation data indicates whether a note is played. Note ensemble data indicates whether a note is played twice consecutively. If a note is reactivated (i.e., replayed) immediately after it has been played, and there is no time interval between the two plays, it is considered that the note has been played twice consecutively. Note velocity indicates the intensity at which the note is played; velocity can also be referred to as volume. Based on this, the note data can be represented as a three-dimensional vector. For example, an ensemble note with a velocity of 0.4 can be represented as the following vector: [1, 1, 0.4]. This vector indicates that the note is played twice consecutively, and the velocity is 0.4.

[0064] Given that polyphonic music may involve multiple notes played simultaneously, multiple vectors are needed to represent chords. Therefore, polyphonic music can be represented as an N*3 matrix, where N represents the number of playable notes in the polyphonic music, representing all potential notes at a given time step. For example, the following matrix represents a chord:

[0065]

[0066] In the matrix above, each row represents a note vector. Within each note vector, the first element is the note activation data, the second element is the note ensemble data, and the third element is the note velocity. It can be seen that this matrix indicates that the first and third notes are being played, and that the velocity of both the first and third notes is 0.5.

[0067] based on Figure 2The neural network model structure shown, for each style type, inputs sample note data and sample image data of that style type into the neural network model, and the neural network model predicts the music data of that style type. Optionally, similar to DeepJ, the neural network model randomly samples from a probability distribution to decide whether to play a note. If a note is chosen to be played, it then samples from another probability distribution to decide whether to play the note again. The dynamics of each note are directly controlled by the neural network model.

[0068] Furthermore, adaptive temperature adjustment techniques can be employed to adjust the temperature parameter of the model's output probability distribution, thereby controlling the balance between diversity and determinism in the output results and preventing the model from generating prolonged silence. For example, if t represents the number of consecutive silent outputs, then the temperature parameter T = 0.1t + 1. This means that whenever the model generates a non-silent output, the temperature parameter T is reset to 1. This adaptive temperature adjustment mechanism can effectively reduce prolonged silent outputs, thereby improving the overall quality of the generated music.

[0069] In this embodiment, the data during model training can be regularized. Optionally, a dropout method can be used to randomly discard some neurons in the network to prevent excessive dependence between neurons and enhance the model's generalization ability. For each set of sample data, neurons in the network can be randomly discarded at a predetermined dropout rate. For example, a 20% dropout rate can be used in the input layer of the model, and a 50% dropout rate can be used in other non-input layers except for the style type embedding layer of image data.

[0070] In some embodiments, the sample note data includes: sample note activation data, sample note ensemble data, and sample note dynamics. The predicted note data includes: predicted note activation data, predicted note ensemble data, and predicted note dynamics.

[0071] During the model training phase, the neural network model generates three distinct outputs: note activation probability, note accompaniment probability, and note dynamics. Therefore, all three outputs need to be trained simultaneously. The note activation probability and note accompaniment probability are treated as logistic regression tasks and trained using binary cross-entropy loss, while the note dynamics are trained using mean squared error. When determining the model loss function for the neural network model based on the predicted note data and sample data, the following steps A1 to A4 can be performed:

[0072] Step A1: Determine the note activation loss function of the neural network model based on the sample note activation data and the predicted note activation data.

[0073] The note activation loss function is used to represent the difference between sample note activation data and predicted note activation data. The note activation loss function can be a binary cross-entropy loss function, expressed as follows:

[0074]

[0075] Among them, L play For the note activation loss function, t play This represents the sample note activation data, y play This represents the predicted note activation data.

[0076] Step A2: Determine the note repetition loss function of the neural network model based on the sample note repetition data and the predicted note repetition data.

[0077] The note encore loss function is used to represent the difference between the sample note encore data and the predicted note encore data. The note encore loss function can be a binary cross-entropy loss function, expressed as follows:

[0078]

[0079] Among them, L replay For the loss function of note repetition, t play t represents the sample note activation data. re Represents sample note ensemble data, y re This represents the predicted note ensemble data.

[0080] Step A3: Determine the note velocity loss function of the neural network model based on the sample note velocity and the predicted note velocity.

[0081] The note velocity loss function is used to represent the difference between the sample note velocity and the predicted note velocity. The note velocity loss function can be a mean squared error loss function, expressed as follows:

[0082]

[0083] Among them, L dynamics Let t be the note velocity loss function. play Indicates the dynamics of the sample note, t dy Represents sample note ensemble data, y dy It indicates the predicted dynamics of a note.

[0084] Step A4: Determine the model loss function based on the note activation loss function, note accompaniment loss function, and note velocity loss function.

[0085] In this embodiment, the timing of the execution of steps A1, A2 and A3 is not limited; they can be executed sequentially or simultaneously.

[0086] Optionally, when performing step A4, the loss weights corresponding to the note activation loss function, note ensemble loss function, and note velocity loss function can be determined first; then, based on the loss weights, the note activation loss function, note ensemble loss function, and note velocity loss function can be weighted and calculated to obtain the model loss function.

[0087] The model loss function L can be expressed as the following formula:

[0088] L=αL play +βL replay +γL dynamics

[0089] Where α, β, and γ represent the loss weights corresponding to the note activation loss function, note ensemble loss function, and note velocity loss function, respectively, and these loss weights are adjustable parameters. Optionally, (α, β, γ) is set to (1, 1, 1).

[0090] Optionally, during step A4, if the predicted note activation data indicates that the note was not activated, then the note activation loss function is set to zero. When the note activation loss function is zero, the model loss function is also set to zero. This is because, for unplayed notes, there is no need to consider ensemble actions and volume.

[0091] Figure 4 A flowchart of the music generation method provided in an embodiment of this application is shown, such as... Figure 4 As shown, the music generation method may include steps S401 and S402.

[0092] Step S401: Obtain note data and image data.

[0093] Step S402: Input the note data and image data into the pre-trained music generation model, classify the image data through the music generation model to obtain the style type corresponding to the image data; generate music based on the note data and style type to obtain the target music that conforms to the style type.

[0094] The music generation model is trained according to the training method of the music generation model in any of the above embodiments.

[0095] According to the technical solution of this application embodiment, note data and image data are acquired; the note data and image data are input into a pre-trained music generation model; the music generation model classifies the image data to obtain the style type corresponding to the image data; music is generated based on the note data and style type to obtain target music that conforms to the style type. Since the music generation model relies not only on note data but also on image data with specific style types during the training phase, note features and image styles simultaneously guide the model's learning objective. Therefore, the trained music generation model can generate high-quality music that matches the style type of a given image, which has high application value for customizing high-quality music for specific visual scenes.

[0096] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a training device for a music generation model.

[0097] Figure 5 A block diagram of the training apparatus for the music generation model provided in this application embodiment is shown, such as... Figure 5 As shown, the training apparatus for the music generation model includes:

[0098] The first acquisition module 51 is used to acquire sample data; the sample data includes: sample note data, sample image data, and label information; the label information is used to indicate the style type corresponding to the sample note data and the sample image data.

[0099] The prediction module 52 is used to input the sample data into the neural network model to be trained to generate music and obtain the predicted note data of the sample data.

[0100] The training module 53 is used to determine the model loss function of the neural network model based on the predicted note data and the sample data, and to iteratively train the neural network model based on the model loss function to obtain a music generation model.

[0101] In some embodiments, when the prediction module 52 inputs the sample data into the neural network model to be trained to generate music and obtains the predicted note data of the sample data, it performs the following steps:

[0102] The sample data is input into the neural network model to be trained, and the style of the sample image data is classified to obtain the predicted style type of the sample image data.

[0103] Based on the sample note data and the predicted style type, the sample note data is predicted to obtain the predicted note data.

[0104] In some embodiments, the predicted note data includes note data for multiple time steps;

[0105] When the prediction module 52 makes a prediction based on the sample note data and the prediction style type, it performs the following steps:

[0106] Extract the note features from the sample note data;

[0107] The note features, the predicted style type, and the note data from the previous time step are fused to obtain fused data;

[0108] Based on the fused data, the note data at the current time step of the sample note data is predicted.

[0109] In some embodiments, the sample note data includes: sample note activation data, sample note ensemble data, and sample note dynamics; the predicted note data includes: predicted note activation data, predicted note ensemble data, and predicted note dynamics.

[0110] When determining the model loss function of the neural network model based on the predicted note data and the sample data, the training module 53 performs the following steps:

[0111] Based on the sample note activation data and the predicted note activation data, the note activation loss function of the neural network model is determined;

[0112] Based on the sample note repetition data and the predicted note repetition data, the note repetition loss function of the neural network model is determined;

[0113] Based on the sample note dynamics and the predicted note dynamics, determine the note dynamics loss function of the neural network model;

[0114] The model loss function is determined based on the note activation loss function, the note ensemble loss function, and the note velocity loss function.

[0115] In some embodiments, when the training module 53 determines the model loss function based on the note activation loss function, the note ensemble loss function, and the note velocity loss function, it performs the following steps:

[0116] Determine the loss weights corresponding to the note activation loss function, the note accompaniment loss function, and the note velocity loss function, respectively;

[0117] Based on the loss weights, the note activation loss function, the note ensemble loss function, and the note velocity loss function are weighted and calculated to obtain the model loss function.

[0118] In some embodiments, when the training module 53 determines the model loss function based on the note activation loss function, the note ensemble loss function, and the note velocity loss function, it performs the following steps:

[0119] If the predicted note activation data indicates that the note is not activated, the value of the note activation loss function is determined to be zero;

[0120] If the value of the note activation loss function is zero, then the value of the model loss function is determined to be zero.

[0121] The apparatus according to embodiments of this application acquires sample data, including sample note data, sample image data, and label information, whereby the label information represents the style type corresponding to the sample note data and sample image data. The sample data is then input into a neural network model to be trained for music generation, resulting in predicted note data. Based on the predicted note data and the sample data, a model loss function for the neural network model is determined, and the model is iteratively trained based on the model loss function to obtain a music generation model. It is evident that the music generation model, during the training phase, relies not only on note data but also on image data with specific style types. This allows note features and image styles to simultaneously guide the model's learning objective, enabling the trained music generation model to generate high-quality music that matches the style type of a given image. This has significant application value for customizing high-quality music for specific visual scenes.

[0122] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0123] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a music generation device.

[0124] Figure 6 A block diagram of a music generation apparatus provided in an embodiment of this application is shown, such as... Figure 6 As shown, the music generation device includes:

[0125] The second acquisition module 61 is used to acquire note data and image data;

[0126] The generation module 62 is used to input the note data and image data into a pre-trained music generation model, classify the image data through the music generation model to obtain the style type corresponding to the image data, and generate music based on the note data and the style type to obtain target music that conforms to the style type.

[0127] The music generation model is trained according to the training method of the music generation model described in any of the above embodiments.

[0128] The apparatus according to embodiments of this application acquires musical note data and image data; inputs the musical note data and image data into a pre-trained music generation model; classifies the image data using the music generation model to obtain the style type corresponding to the image data; and generates music based on the musical note data and style type to obtain target music that conforms to the style type. Since the music generation model relies not only on musical note data but also on image data with specific style types during the training phase, both musical note features and image styles simultaneously guide the model's learning objective. Therefore, the trained music generation model can generate high-quality music that matches the style type of a given image, which has high application value for customizing high-quality music for specific visual scenes.

[0129] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0130] Figure 7 This is a block diagram for implementing the electronic device provided in the embodiments of this application. Figure 7 As shown, the electronic device includes a memory 701 and a processor 702. The memory 701 stores a computer program that can run on the processor 702. When the processor 702 executes the computer program, it implements the method described in the above embodiments. The number of memories 701 and processors 702 can be one or more. In a specific implementation, the electronic device may also include a communication interface 703 for communicating with external devices and performing data exchange and transmission.

[0131] In practical implementation, if the memory 701, processor 702, and communication interface 703 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0132] Optionally, in a specific implementation, if the memory 701, processor 702, and communication interface 703 are integrated on a single chip, the memory 701, processor 702, and communication interface 703 can communicate with each other through an internal interface.

[0133] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0134] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0135] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0136] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0137] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0138] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0139] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0140] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0141] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0142] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0143] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0144] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0145] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0146] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A training method for a music generation model, characterized in that, include: Obtain sample data; The sample data includes: sample note data, sample image data, and label information; the label information is used to indicate the style type corresponding to the sample note data and the sample image data. The sample data is input into the neural network model to be trained to generate music, and the predicted note data of the sample data is obtained. Based on the predicted note data and the sample data, the model loss function of the neural network model is determined, and the neural network model is iteratively trained based on the model loss function to obtain a music generation model.

2. The method according to claim 1, characterized in that, The step of inputting the sample data into the neural network model to be trained for music generation, to obtain the predicted note data of the sample data, includes: The sample data is input into the neural network model to be trained, and the style of the sample image data is classified to obtain the predicted style type of the sample image data. Based on the sample note data and the predicted style type, the sample note data is predicted to obtain the predicted note data.

3. The method according to claim 2, characterized in that, The predicted note data includes note data at multiple time steps; The step of predicting the sample note data based on the sample note data and the predicted style type includes: Extract the note features from the sample note data; The note features, the predicted style type, and the note data from the previous time step are fused to obtain fused data; Based on the fused data, the note data at the current time step of the sample note data is predicted.

4. The method according to claim 1, characterized in that, The sample note data includes: sample note activation data, sample note ensemble data, and sample note dynamics; the predicted note data includes: predicted note activation data, predicted note ensemble data, and predicted note dynamics. The step of determining the model loss function of the neural network model based on the predicted note data and the sample data includes: Based on the sample note activation data and the predicted note activation data, the note activation loss function of the neural network model is determined; Based on the sample note repetition data and the predicted note repetition data, the note repetition loss function of the neural network model is determined; Based on the sample note dynamics and the predicted note dynamics, determine the note dynamics loss function of the neural network model; The model loss function is determined based on the note activation loss function, the note ensemble loss function, and the note velocity loss function.

5. The method according to claim 4, characterized in that, The step of determining the model loss function based on the note activation loss function, the note accompaniment loss function, and the note velocity loss function includes: Determine the loss weights corresponding to the note activation loss function, the note accompaniment loss function, and the note velocity loss function, respectively; Based on the loss weights, the note activation loss function, the note ensemble loss function, and the note velocity loss function are weighted and calculated to obtain the model loss function.

6. The method according to claim 4, characterized in that, The step of determining the model loss function based on the note activation loss function, the note accompaniment loss function, and the note velocity loss function includes: If the predicted note activation data indicates that the note is not activated, the value of the note activation loss function is determined to be zero; If the value of the note activation loss function is zero, then the value of the model loss function is determined to be zero.

7. A method for generating music, characterized in that, include: Acquire note data and image data; The note data and image data are input into a pre-trained music generation model. The music generation model classifies the image data to obtain the style type corresponding to the image data. Music is generated based on the note data and the style type to obtain target music that conforms to the style type. The music generation model is trained using the training method for music generation models according to any one of claims 1 to 6.

8. A training device for a music generation model, characterized in that, include: The first acquisition module is used to acquire sample data; The sample data includes: sample note data, sample image data, and label information; the label information is used to indicate the style type corresponding to the sample note data and the sample image data. The prediction module is used to input the sample data into the neural network model to be trained to generate music and obtain the predicted note data of the sample data. The training module is used to determine the model loss function of the neural network model based on the predicted note data and the sample data, and to iteratively train the neural network model based on the model loss function to obtain a music generation model.

9. A music generation device, characterized in that, include: The second acquisition module is used to acquire note data and image data; The generation module is used to input the note data and image data into a pre-trained music generation model, classify the image data through the music generation model to obtain the style type corresponding to the image data, and generate music based on the note data and the style type to obtain target music that conforms to the style type. The music generation model is trained using the training method for music generation models according to any one of claims 1 to 6.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 7.

Images