Controllable generation method and system of symbol music, electronic equipment and medium

Abstract

The invention provides a controllable generation method and system for symbolic music, electronic equipment and a computer readable storage medium, and the method comprises the steps: carrying out the preprocessing of the pre-obtained music content and annotation information through a preset representation module, so as to form an original data set; performing masking processing on the original data set according to a preset chord condition rule to form an original masking data pair, and sending the original masking data pair into a preset music discrete diffusion framework for repeated training to form a music discrete diffusion model; and then performing preorder processing on the music frequency band and the music demand input by the user to form a music content representation and condition matrix, inputting the music content representation and condition matrix into the music discrete diffusion model, and enabling the music discrete diffusion model to perform music generation so as to output a target music score. The richness of music controllable generation can be realized, and the reliability and rationality can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more specifically, to a method, system, electronic device and medium for controllable generation of symbolic music. Background Art

[0002] Traditional music creation requires producers to have professional music theory knowledge and rich production experience, and has high professional thresholds and artistic literacy requirements. With the rapid development of artificial intelligence and multimedia technology, as well as the gradual development of AI-generated content, symbolic music (especially computer music in MIDI format) can also realize intelligent composition, intelligent arrangement, and other aspects to assist human music creation, becoming a music production tool. However, the existing symbolic music generation technology is slightly lacking in the harmony and rationality of multi-track and multi-instrument music generation content, resulting in poor quality of generated content, which cannot meet users' demand for high-quality music content.

[0003] For example, Microsoft Pop-MAG proposed Multi-Track MIDI Representation (MTRepresentation), which re-encodes the word sequence into a two-dimensional sequence according to event attributes, providing a way of organizing data. However, when faced with multi-track and multi-instrument music, the model does not learn the note distribution characteristics of various instruments well, and the rationality of the instruments in the generated music is poor.

[0004] Microsoft Pop-MAG is a classic accompaniment generation work based on the Transformer method. These classic generative models require the decoder network to pay attention to the correct conditional token or hidden layer feature vector when predicting the next sentence. Obtaining high-quality samples depends on a larger model size, and as the sentence becomes longer, the conditional constraints become weaker, which makes the decoder-based autoregressive model encounter bottlenecks in generation quality and controllable generation issues.

[0005] SCHmUBERT is a discrete diffusion model for music generation, but it requires a trained classifier and an additional convolutional neural network, and is unconditional accompaniment generation. Its public work can only handle bass and drum accompaniment. There are still problems to be solved in terms of the controllable generation ability of probabilistic diffusion models.

[0006] Therefore, there is an urgent need for a controllable generation scheme of symbolic music that can improve the generation quality of multi-track symbolic music and enhance the harmony and rationality of multi-track music. Summary of the invention

[0007] In view of the above problems, the purpose of the present invention is to provide a controllable generation method, system, electronic device and medium of symbolic music, so as to solve the technical problem that the existing symbolic music generation technology lacks the harmony and rationality of the multi-track and multi-instrument music generation content, resulting in poor quality of generated content and unable to meet users' demand for high-quality music content.

[0008] In a first aspect, an embodiment of the present application provides a controllable method for generating symbolic music, the controllable method for generating symbolic music comprising: Preprocessing the pre-acquired music content and annotation information through a preset representation module to form an original data set; Performing masking processing on the original data set according to a preset chord condition rule to form an original masked data pair; Sending the original masked data pair into a preset music discrete diffusion framework for repeated training to form a music discrete diffusion model; The music frequency band and music demand input by the user are pre-processed to form a music content representation and a condition matrix, and the music content representation and the condition matrix are input into the music discrete diffusion model, so that the music discrete diffusion model performs music generation to output a target music score.

[0009] Optionally, the pre-processing of the music frequency band and music requirement input by the user to form a music content representation and a condition matrix includes: Segmenting the music frequency band to form music clips; Each music clip is encoded to form a music content representation; element information of the music clip is obtained, the element information is combined with the music requirement to obtain music combination information, and the music combination information is encoded to form a condition matrix.

[0010] Optionally, the music discrete diffusion model performs music generation to output a target music score, including: The music content representation and condition matrix of each music clip are concatenated to form a merged representation; Using a preset three-dimensional word embedding space to perform word embedding of the combined representation in a continuous domain to form a combined word embedding vector; Using a preset layer normalization module to perform Markov shift on the merged word embedding vector to obtain a noise initial state of the merged word embedding vector; Performing dynamic noise addition processing on the masked part in the initial noise state through a preset diffusion module to generate a hidden state vector; Applying position word embedding and time step word embedding to the hidden state vector to generate a diffuse hidden vector, and then performing an inverse operation on the diffuse hidden vector through a preset bidirectional encoder network to obtain a target hidden vector; Performing category mapping on the target hidden vector through a preset multi-head output module to obtain a target music content representation matrix; Decoding the target music content representation matrix to generate a target music score frequency band; All target music score frequency bands are concatenated in chronological order to form a target music score.

[0011] Optionally, a music sampling file is loaded into the target music score through a preset audio library to generate target music.

[0012] Optionally, the preprocessing of the pre-acquired music content and annotation information by a preset characterization module to form an original data set includes: The chords of the music content are used as a timeline, the start time of the notes is automatically quantized in units of 32-notes, the duration of the notes is quantized in units of 16-notes, self-correction is performed, and empty tracks and empty notes are self-deleted to form standard music content, and the standard music content is self-annotated using the annotation information to form a music content set; wherein the music content set contains various music events of the standard music content, and the music events are annotated with music attributes; Combining all music events of a piece of music content into a matrix to form a music content representation, and combining the music content representations of all the music contents to form a music content representation data set; The music content representation dataset is used as the original dataset.

[0013] Optionally, performing masking processing on the original data set according to a preset chord condition rule to form an original masked data pair includes: The chord music events represented by the music content in the original data set are displayed, and the non-chord music events represented by the music content are masked; wherein the masked music events are marked with PAD.

[0014] Optionally, in the process of sending the original masked data pair into a preset music discrete diffusion framework for repeated training to form a music discrete diffusion model, The music discrete framework is trained using original masked data through normalized embedding learning; wherein the music discrete framework includes a diffusion model framework, a decoding model framework and a word embedding module; The norm loss of the diffusion model framework is calculated by a preset BERT model, and the attention mechanism of the diffusion model framework is data-dependent with the word embedding vector involved in the word embedding module; The objective function when training the music discrete diffusion framework is the sum of the loss function of the diffusion model framework, the loss function of the decoding model framework and the loss function of the word embedding module.

[0015] In a second aspect, the embodiment of the present application provides a controllable generation system of symbolic music, which implements the controllable generation method of symbolic music as described above, and the system includes a music discrete diffusion model; wherein: The music discrete diffusion model is generated by learning through repeated training of original masked data pairs using a preset music discrete diffusion framework; wherein obtaining the original masked data pairs includes: preprocessing the pre-acquired music content and annotation information through a preset representation module to form an original data set; performing masking processing on the original data set according to a preset chord condition rule to form an original masked data pair; The music discrete diffusion model is used to receive the music content representation and condition matrix formed by pre-processing the music frequency band and music requirements input by the user, and generate music according to the music content representation and condition matrix to output the target music score.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a system bus; the processor and the memory are connected via the system bus; the memory is used to store one or more programs, the one or more programs comprising instructions, which when executed by the processor enable the processor to execute any one of the steps in the controllable generation method of symbolic music described in the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any optional method for controllable generation of symbolic music as described in the first aspect.

[0018] It can be seen from the above technical solutions that the embodiments of the present application provide a controllable generation method, system, electronic device and computer-readable storage medium for symbolic music. Compared with the prior art, the present application has the following beneficial effects: Using the chords of the musical content as a timeline can significantly reduce the sequence length and preserve temporal and multi-track spatial information; The original data set is masked according to a preset chord condition rule to form an original masked data pair, thereby strengthening the restriction of chords in composition and improving the effectiveness of feature learning and generation of popular music; When training and using the music discrete diffusion model, the matrix is ​​first concatenated, then discrete noise is added, and finally decoding is performed. After decoding, the probability mapping of the hidden vector is also required. This can not only achieve the richness of controllable music generation, but also improve reliability and rationality. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] By referring to the following description in conjunction with the accompanying drawings, and with a more comprehensive understanding of the present invention, other objects and results of the present invention will become more clear and easy to understand. In the accompanying drawings: Figure 1 is a flow chart of a controllable method for generating symbolic music according to an embodiment of the present invention; Figure 2 A schematic diagram of a use scenario of a controllable method for generating symbolic music according to an embodiment of the present invention; Figure 3 A schematic diagram of a music discrete diffusion model of a controllable method for generating symbolic music according to an embodiment of the present invention; Figure 4 A schematic diagram of a three-dimensional word embedding space of a controllable generation method of symbolic music according to an embodiment of the present invention; Figure 5 A schematic diagram of a multi-head output module of a controllable method for generating symbolic music according to an embodiment of the present invention; Figure 6 A logic block diagram of a controllable generation system of symbolic music according to an embodiment of the present invention; Figure 7 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The existing symbolic music generation technology lacks the harmony and rationality of the generated content of multi-track and multi-instrument music, resulting in poor quality of the generated content and failing to meet users' technical needs for high-quality music content.

[0021] In view of the above problems, the present invention provides a controllable generation method, system, electronic device and computer-readable storage medium of symbolic music, firstly, the pre-acquired music content and annotation information are pre-processed by a preset representation module to form an original data set, the original data set is masked according to the preset chord condition rule to form an original masked data pair, and then the original masked data pair is sent to a preset music discrete diffusion framework for repeated training to form a music discrete diffusion model, and then the music frequency band and music demand input by the user are pre-processed to form a music content representation and a condition matrix, and the music content representation and the condition matrix are input into the music discrete diffusion framework. In the scattered model, the music discrete diffusion model is used to generate music to output the target music score. In this way, the chords of the music content are used as the timeline, which can significantly reduce the sequence length and retain the time and multi-track space information. The original data set is masked according to the preset chord condition rules to form the original masked data pair, which strengthens the restrictions on chords in composition and improves the effectiveness of feature learning and generation of popular music. When training and using the music discrete diffusion model, the matrix is ​​first spliced, then discrete denoising is performed, and finally decoding is performed. After decoding, the probability mapping of the hidden vector is also required, which can not only achieve the richness of controllable music generation, but also improve reliability and rationality.

[0022] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. The description of the following exemplary embodiments is actually merely illustrative and is by no means intended to limit the present invention and its application or use. The techniques and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, the techniques and equipment should be considered as part of the specification. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0023] Figure 1 A flowchart of a controllable method for generating symbolic music provided in an embodiment of the present application, such as Figure 1 As shown, the method includes: S1: preprocessing the pre-acquired music content and annotation information through a preset representation module to form an original data set; S2: performing masking processing on the original data set according to a preset chord condition rule to form an original masked data pair; S3: sending the original masked data pair into a preset music discrete diffusion framework for repeated training to form a music discrete diffusion model; S4: Pre-process the music frequency band and music demand input by the user to form a music content representation and a condition matrix, input the music content representation and the condition matrix into the music discrete diffusion model, and enable the music discrete diffusion model to generate music to output a target music score.

[0024] Step S1 is a process of preprocessing the pre-acquired music content and annotation information through a preset representation module to form an original data set. This process is a basic step for training the subsequent music discrete diffusion model. The process of preprocessing the pre-acquired music content and annotation information through a preset representation module to form an original data set includes: S11: using the chords of the music content as a timeline, automatically quantizing the start time of the notes in units of 32-notes, quantizing the duration of the notes in units of 16-notes, performing self-correction, and self-deleting empty tracks and empty notes to form standard music content, and using the annotation information to self-annotate the standard music content to form a music content set; wherein the music content set contains various music events of the standard music content, and the music events are annotated with music attributes; S12: Matrix-combining all music events of a piece of music content to form a music content representation, and combining the music content representations of all the music contents to form a music content representation data set; S13: Use the music content representation dataset as the original dataset.

[0025] In a specific embodiment of the present invention, the miditoolkit tool of Python can be used to read music content information from a Musical Instrument Digital Interface (MIDI) file (a music file with a .mid extension), and jsonl can be used to read the text annotation information corresponding to the music from the annotation file. The annotation information includes the style and mood of the music. We extracted a total of about 6,000 pairs of MIDI music and annotations.

[0026] Since the music content recorded in MIDI files often has problems such as misaligned note beats, incorrect editing of note duration, empty tracks, etc., and MIDI files usually do not have instrument type labels. Therefore, in this specific embodiment, MIDI files are preprocessed first, such as: (1) automatically quantizing the start time of notes in units of 32nd notes; (2) quantizing the note duration in units of 16th notes; (3) automatically correcting errors and deleting empty tracks and empty notes; (4) automatically annotating instrument types according to the MIDI protocol, etc.

[0027] In symbolic music, notes are the most basic elements, which are described by note attributes such as instrument type (ins), attack time (time), pitch (pitch) and duration (dur). ij The jth attribute of the i-th note is represented. The chord is regarded as a special note, and the chord type (such as the main chord, the augmented fourth chord) is classified as a pitch event. In this specific embodiment, the music information other than the chord and the note is collectively referred to as a meta event, such as the music speed (Tempo), the measure number mark, and the track change mark.

[0028] In this specific embodiment, the chord condition rule is adopted, which includes two parts: relative time coding and a masking method based on the chord condition.

[0029] The relative time encoding of the chord conditional rule is to use chords to calculate the relative time information of the music content, that is, the chord progression is the timeline of the entire music content, rather than using meta-information tags (bars, tracks) like SymphonyNet and other REMI works. In this embodiment, the chords of the music content are used as the timeline method, which can significantly reduce the sequence length and retain the time and multi-track space information. Explanation: (1) Bar number marks and track change marks are used to record the score and have little effect on auditory appreciation. (2) For modern pop music, all tracks follow a chord progression in auditory perception, and the entire music also follows the development of the chord progression. Strengthening the compositional constraints of chords is very effective for feature learning and generation of pop music.

[0030] Therefore, in this embodiment, the attribute event e is used ij The music content is tokenized, and other important meta-events, such as music speed, are introduced into the training and generation process as conditions, just like emotion and style labels.

[0031] The word sequence must conform to the correct encoding grammatical structure. For example, according to the customized encoding scheme, the time event should be after the pitch, otherwise the generated word sequence cannot be correctly decoded into a MIDI file. According to repeated experiments, using a language model to train a REMI-like word sequence representation that is not restricted by grammatical structure is very unstable.

[0032] Therefore, in this embodiment, a structure-aligned multi-track multi-instrument representation is adopted, with I x J music events e ij (i=1,2,...,I ​​j=1,...,J) form the matrix W=[e ij ] to express the content of a piece of symbolic music. In the experiment, the event e i1 =insi ,e i2 =time i, e i3 =pit i, e i4 =dur i represents the instrument attribute, start time attribute, pitch attribute and duration attribute of the i-th note in turn, so the music representation W can be expressed as: Formula (1) The details of the music representation are designed as follows: (1) ins i ins i Indicates the melody, chord label and instrument type of the i-th note. Notes belonging to the same instrument are merged into one track. Chords are extracted using the open source algorithm of the Compound word transformer. In this embodiment, all 16 instrument categories are used according to the MIDI protocol, and guitars are divided into acoustic guitars and electric guitars, and chords and labels, melody labels and unknown labels are added, for a total of 20 categories. All existing instruments and sounds are classified into these 20 categories, and more specific categories are shown in the following table; Table 1 Instrument types

[0033] (2) time i The attack time of the ith note (chord) relative to the previous chord, in thirty-second notes, ranging from 0 to 64 (up to 2 bars in 4 / 4 time signature).

[0034] (3) pit i The pitch of the ith note, ranging from C1 to B6, or the type of the ith chord. Includes 12 roots and 9 types.

[0035] (4) dur i The duration of the ith note, in sixteenth notes, ranging from 1 to 16 (maximum 1 measure in 4 / 4 time signature). Chord events fill the “PAD” markers.

[0036] Chord progressions naturally form a collection of time intervals. Based on the semi-permutation invariant of multi-track music, we arrange and flatten the note units within each chord time interval into a sequence. This flattening method combined with a bidirectional encoder model can naturally establish harmony between multiple tracks, and each instrument category can learn its note distribution characteristics.

[0037] Finally, according to the above method, the music content information in the MIDI file is encoded into a music content representation matrix, named Wm.

[0038] Step S2 is a process of performing masking processing on the original data set according to a preset chord condition rule to form an original masked data pair, wherein the masking processing on the original data set according to the preset chord condition rule to form an original masked data pair includes: The chord music events represented by the music content in the original data set are displayed, and the non-chord music events represented by the music content are masked; wherein the masked music events are marked with PAD.

[0039] For example, in a specific embodiment, for a specific task, in W m A dynamic event-level masking strategy is used, that is, for the music events that need to be learned and predicted, ij Masking is implemented, and other events are displayed. The events that need to be displayed are called conditional events and are displayed normally. This process is completed by the trained masked language model (MLM), or more precisely, the masked noise model is trained in the diffusion framework to reconstruct the masked event information. If there is no music event at a certain position, it is predicted as a PAD label.

[0040] In this embodiment, the chord condition rule proposed above is adopted: during the training process, W m The chord events in the , thus making the chord events a condition. When training the music discrete diffusion model and subsequently using the music discrete diffusion model, the masking strategy for text-to-symbol music generation is as shown in formula (2), W c Full display, W m Chord events in the matrix are displayed and other events are masked.

[0041] Formula (2) is: W c =[ label Tempo_Andante jazz ... form VERSE fragment_1 ] (show all) W m =[ [ins chord , time1, pit1,PAD], (the 4 events in this row are chord related events, all are displayed) [ins melody , time2, pit2, dur2], (all 4 events in this row are masked) [ins3, time3, pit3, dur3], (all 4 events in this row are masked) ... [ins chord , time i , pit i ,PAD], (The 4 events in this row are chord related events, all are displayed) [ins i+1 , time i+1 , pit i+1 ,dur i+1 ], (all 4 events in this row are masked) ... ] It should be noted that if the first word in this line is inschord, the four events in this line are chord-related events and are all displayed; otherwise, the four events in this line are masked; the masked events are set as PAD markers. According to the method proposed in this embodiment, as long as a specific masking strategy is implemented, any desired instrument and event can be generated under the control of given information. ij The greatest value of this representation and masking method is to support human-computer interaction of artificial intelligence generated content (AIGC). The specific interaction methods are as follows: Figure 2 As shown, for example, the user can propose to upload a missing musical note and input the user's requirements, and the method in this embodiment will generate suitable music to fill the gap according to the device requirements; the user can also propose modification requirements for the original complete musical note, or even just input a piece of text, and the system will automatically match the corresponding musical note to generate music.

[0042] Step S3 is a process of sending the original masked data pair into a preset music discrete diffusion framework for repeated training to form a music discrete diffusion model; wherein, in the process of sending the original masked data pair into a preset music discrete diffusion framework for repeated training to form a music discrete diffusion model, S31: training the music discrete framework using original masked data through normalized embedding learning; wherein the music discrete framework includes a diffusion model framework, a decoding model framework and a word embedding module; S32: the norm loss of the diffusion model framework is calculated by a preset BERT model, and the attention mechanism of the diffusion model framework is data-related to the word embedding vector involved in the word embedding module; S33: The objective function when training the music discrete diffusion framework is the sum of the loss function of the diffusion model framework, the loss function of the decoding model framework and the loss function of the word embedding module.

[0043] In one embodiment, the variational lower limit L is derived according to the standard diffusion process VLB The first difference between this method and the existing technology is the normalized embedding learning, that is, the use of anchor loss ~\cite{gao2022difformer} instead of the round loss of Diffuseq in the existing technology to further alleviate the collapse of the denoising target. The anchor loss uses the predicted value of the model after training. As input, this makes the word embeddings more informative and more distinguishable from each other. Second, unlike Difformer, the optimization term of the decoding process is still retained, named L nll .

[0044] In order to cooperate with the masked denoising process, the final objective function is shown in formula (3): ————Formula (3); in, represents the minimization of the variational lower bound, Represents the given variable Minimize. It is the abbreviation of variational lower bound; Z0 represents the initial state; Represents the state of step t; represents the non-classifier discrete diffusion model; Represents the hidden state generated (output) by the non-classifier discrete diffusion model; stands for three-dimensional word embedding; Represents the parameters The conditional probability of Represents the parameters In giving input In the case of The conditional probability of Represents the representation matrix.

[0045] The first one guides the diffusion process of the diffusion model framework, the second one is responsible for the decoder process of the decoding model framework, and the third one uses anchor loss to train the word embedding module. The first one (L2 norm loss) is calculated by BERT, and its attention mechanism also takes into account the displayed word embedding vector. Therefore, the back-propagated gradient affects the correlation learning with the conditional and generated vectors and jointly optimizes the word embedding module.

[0046] More specifically, the work performed by the model during the training process is the same as the workflow run by the following trained music discrete diffusion model when generating music based on the user's music content representation and conditional matrix.

[0047] Step S4 is a process of pre-processing the music frequency band and music demand input by the user to form a music content representation and a condition matrix, inputting the music content representation and the condition matrix into the music discrete diffusion model, and making the music discrete diffusion model generate music to output the target music score; that is, the steps performed during training can refer to the steps when the trained music discrete diffusion model is applied, as shown in the following steps S421-S428 and their embodiments.

[0048] The pre-processing of the music frequency band and music requirements input by the user to form a music content representation and a condition matrix includes: S411: Segment the music frequency band to form music clips; S412: Encode each music clip to form a music content representation; obtain element information of the music clip, combine the element information with the music requirement to obtain music combination information, and encode the music combination information to form a conditional matrix.

[0049] The music discrete diffusion model performs music generation to output a target music score, including: S421: splicing the music content representation and the condition matrix of each music clip to form a combined representation; S422: Using a preset three-dimensional word embedding space to perform word embedding of the combined representation in a continuous domain to form a combined word embedding vector; S423: Using a preset layer normalization module to perform Markov transfer on the merged word embedding vector to obtain a noise initial state of the merged word embedding vector; S424: dynamically adding noise to the masked part of the noise initial state through a preset diffusion module to generate a hidden state vector; S425: applying position word embedding and time step word embedding to the hidden state vector to generate a diffuse hidden vector, and then performing an inverse operation on the diffuse hidden vector through a preset bidirectional encoder network to obtain a target hidden vector; S426: performing category mapping on the target hidden vector through a preset multi-head output module to obtain a target music content representation matrix; S427: Decoding the target music content representation matrix to generate a target music score frequency band; S428: splicing all target music score frequency bands in chronological order to form a target music score.

[0050] Figure 3 A music discrete diffusion model is shown, which includes a discrete diffusion module and a deep learning framework for symbolic music generation. Figure 3 The left part of is the diffusion process, in which the events that need to be learned and generated are masked and participate in diffusion and reconstruction, while the conditional events (such as chord events and rhythm events) remain unchanged; Figure 3 The right part of is the encoder framework, which is a parameterized model of the discrete diffusion framework.

[0051] In a specific implementation, as shown in the left side of FIG3 , the Markov process composed of circular nodes is a standard Denoising Diffusion Probabilistic Models (DDPM), and other modules are the conversion process between discrete and continuous spaces involved in the diffusion process.

[0052] In order to achieve parallel generation of long segments, the music score is divided into 120 notes as a segment, and the music content representation matrix W of each segment is obtained: m n , and record the fragment identifier fragment_n in the corresponding Wc, where n is the serial number of the identified fragment. First, you need to splice Wc and W m n , thereby obtaining the combined representation W of each fragment c+m n； Afterwards, use the following Figure 4 The learnable three-dimensional word embedding module shown in Figure 2 represents the combined discrete domain W. c+m n Mapped to the continuous domain word embedding space EMB (W c+m n ) to form a merged word embedding vector. Figure 4 As shown, in this embodiment, the size (Size) of the word embedding space is designed to be [I, J, D], where I represents a note plus a description phrase, J represents an attribute, and D represents a word embedding vector. I and J are interchangeable.

[0053] In musical scores, the frequencies of common and rare events are extremely unbalanced, which has an adverse impact on the diversity of content generated by the model. To solve this problem, in this implementation, a learnable layer normalization module (named LN) is used on the word embedding space to input discrete event information into the initial state Z0 of the diffusion process by extending the standard forward chain into a new Markov transfer process.

[0054] That is, according to the specific task W mThe mask positions are arranged where noise is applied dynamically only to the mask vector of Zt, and the displayed portion of Zt remains unchanged throughout the diffusion process.

[0055] Then attach Figure 3 The process of converting the discrete part of the noise into a discrete part is to dynamically add noise to the masked part of the initial noise state through a preset diffusion module to generate a hidden state vector, that is, the process of converting Z0 into Zt; this process can train a non-classifier discrete diffusion model in advance To complete the inverse process, the applied formula (4) is as follows: Formula (4) in and In the forward process parameterization of the predicted mean and standard deviation of ; is the predicted mean, is the standard deviation of the prediction, is the state at step t. is the conditional probability of the forward process.

[0056] Then, position word embedding and time step word embedding are applied to the hidden state vector to generate a diffuse hidden vector, and then the diffuse hidden vector is inversely operated through a preset bidirectional encoder network to obtain a target hidden vector. The bidirectional encoder network (BERT) can be used to simulate , by gradually converting arrive , and then to the final goal Complete the inverse operation process to obtain the target hidden vector, that is, the target .

[0057] Finally, the target hidden vector is mapped to a category by a preset multi-head output module to obtain a target music content representation matrix. The multi-head output module is shown in the attached figure. Figure 5 As shown, it is connected to the above-mentioned Figure 3 At the end of the framework, the Multi-Head output module is used to map the encoder hidden state (target hidden vector) to a certain attribute category with absolute accuracy. For example, the pitch dimension (W m n The hidden vectors in the third column of the graph must be mapped to one of the 72 pitch categories and cannot be classified as onset or other categories. Figure 5As shown in Figure 5, it consists of multiple linear layers and classification masks. The linear layer (the cube part in Figure 5) uses a fully connected network to convert the hidden state vector output by BERT into a probability sequence with the same length as the dictionary (vocabulary size), which reflects the probability score of each word (Token). This specific embodiment divides the hidden state tensor into 4 blocks by column (attribute) and arranges 4 linear layers to predict their respective probability scores. Conditional information W c Concatenate with the first column (instrument attribute) and input to the first linear layer. Then the obtained probability sequence is bitwise multiplied by the 0-1 mask sequence Mask j , the token position with the probability score sequence set to 0 will not be selected, and the prediction content with high probability will be connected according to the grammatical structure of music representation to obtain the target music content representation matrix generated by the model .

[0058] The target music content representation matrix is ​​then decoded to generate target music score frequency bands, and then all target music score frequency bands are spliced ​​according to time to form a target music score.

[0059] Since the generated target music score is MIDI, which is a music score without sound, in this specific embodiment, the Fluidsynth library is used to load the music sampling file in SoundFont2 format for the target music score, thereby exporting the music audio and saving and downloading it as a WAV file.

[0060] Therefore, the trained music discrete diffusion model is used to generate music and output the target music score.

[0061] The training process of the music discrete diffusion model is similar to the use process of the above-mentioned music discrete diffusion model. It only requires replacing the music frequency bands and music requirements input by the above-mentioned user with the above-mentioned original data set and its original masked data pairs, which will not be elaborated here.

[0062] As described above, the controllable generation method of symbolic music provided by the present invention uses the chords of the music content as the timeline, which can significantly reduce the sequence length and retain the time and multi-track space information; the original data set is masked according to the preset chord condition rules to form an original masked data pair, strengthens the restrictions on chords in composition, and improves the effectiveness of feature learning and generation of popular music; when training and using the music discrete diffusion model, the matrix is ​​first spliced, then discrete noise is added, and finally decoding is performed. After decoding, probability mapping of the hidden vector is also required, which can not only achieve the richness of controllable generation of music, but also improve reliability and rationality.

[0063] A controllable generation system 100 of symbolic music provided in an embodiment of the present application is introduced below. The controllable generation system of symbolic music described below and the controllable generation method of symbolic music described above can refer to each other.

[0064] like Figure 6 As shown, the present invention also provides a controllable generation system 100 of symbolic music, which can implement the controllable generation method of symbolic music as described above, and the system includes: A music discrete diffusion model 101; wherein the music discrete diffusion model 101 is generated by learning through repeated training of original masked data pairs using a preset music discrete diffusion framework; wherein obtaining the original masked data pairs comprises: preprocessing the pre-acquired music content and annotation information through a preset representation module to form an original data set; and performing masking processing on the original data set according to a preset chord condition rule to form an original masked data pair; The music discrete diffusion model 101 is used to receive the music content representation and condition matrix formed by pre-processing the music frequency band and music demand input by the user, and generate music according to the music content representation and condition matrix to output the target music score.

[0065] The system further includes a pre-processing unit 102, which is used to perform pre-processing on the music frequency band and music requirements input by the user to form a music content representation and a condition matrix, including: Segmenting the music frequency band to form music clips; Each music clip is encoded to form a music content representation; element information of the music clip is obtained, the element information is combined with the music requirement to obtain music combination information, and the music combination information is encoded to form a condition matrix.

[0066] The controllable generation system 100 of symbolic music provided by the present invention generally performs calculations as the controllable generation method of symbolic music described above. For a more specific implementation process, reference may be made to the specific embodiments of the controllable generation method of symbolic music described above.

[0067] As described above, the controllable generation system 100 of symbolic music provided by the present invention receives the music content representation and condition matrix formed by pre-processing the music frequency bands and music requirements input by the user through the music discrete diffusion model, and generates music according to the music content representation and condition matrix to output the target music score, thereby improving the reliability of symbolic music generation.

[0068] See also Figure 7, which is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, including: a memory 11 for storing a computer program; a processor 12 for implementing the steps of the controllable generation method of symbolic music described in any of the above method embodiments when executing the computer program. In this embodiment, the device can be a vehicle-mounted computer, a PC (Personal Computer), or a terminal device such as a smart phone, a tablet computer, a handheld computer, a portable computer, etc. The device may include a memory 11, a processor 12, and a bus 13. Among them, the memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the device, such as a hard disk of the device. In other embodiments, the memory 11 may also be an external storage device of the device, such as a plug-in hard disk equipped on the device, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. Further, the memory 11 may also include both an internal storage unit of the device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the device, such as program codes for executing fault prediction methods, but also can be used to temporarily store data that has been output or is to be output. The processor 12 may be a central processing unit (CPU) in some embodiments. The processor 12 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip in some embodiments, used to run program codes or process data stored in the memory 11, such as codes for executing a controllable generation method of symbolic music. The bus 13 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 In the figure, only one thick line is used, but it does not mean that there is only one bus or one type of bus. Further, the device may also include a network interface 14, which may optionally include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used to establish a communication connection between the device and other electronic devices.

[0069] Optionally, the device may further include a user interface 15, which may include a display (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 15 may also include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the device and to display a visual user interface. It will be understood by those skilled in the art that Figure 7 The structure shown does not constitute a limitation of the device, and may include fewer or more components than shown, or combine certain components, or arrange the components differently.

[0070] An embodiment of the present invention further provides a computer-readable storage medium, which may be non-volatile or volatile, and stores a computer program, which, when executed by a processor, implements: Preprocessing the pre-acquired music content and annotation information through a preset representation module to form an original data set; Performing masking processing on the original data set according to a preset chord condition rule to form an original masked data pair; Sending the original masked data pair into a preset music discrete diffusion framework for repeated training to form a music discrete diffusion model; The music frequency band and music demand input by the user are pre-processed to form a music content representation and a condition matrix, and the music content representation and the condition matrix are input into the music discrete diffusion model, so that the music discrete diffusion model performs music generation to output a target music score.

[0071] Specifically, the specific implementation method when the computer program is executed by the processor can refer to the description of the relevant steps in the controllable generation method of symbolic music in the embodiment, which will not be repeated here.

[0072] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0073] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0074] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0075] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0076] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0077] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0078] It should be noted that each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for methods, devices, electronic devices and computer-readable storage media, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments. The methods, devices, electronic devices and media described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components indicated as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0079] As described above with reference to the accompanying drawings, the controllable generation method, system, electronic device and medium of symbolic music proposed in the present invention are described by way of example. However, it should be understood by those skilled in the art that various improvements can be made to the controllable generation method, system, electronic device and medium of symbolic music proposed in the present invention without departing from the content of the present invention. Therefore, the protection scope of the present invention should be determined by the content of the attached claims.

Claims

1. A controllable method for generating symbolic music, characterized in that: include: Preprocessing the pre-acquired music content and annotation information through a preset representation module to form an original data set; Performing masking processing on the original data set according to a preset chord condition rule to form an original masked data pair; Sending the original masked data pair into a preset music discrete diffusion framework for repeated training to form a music discrete diffusion model; The music frequency band and music demand input by the user are pre-processed to form a music content representation and a condition matrix, and the music content representation and the condition matrix are input into the music discrete diffusion model, so that the music discrete diffusion model performs music generation to output a target music score.

2. The controllable generation method of symbolic music according to claim 1, characterized in that: The pre-processing of the music frequency band and music requirements input by the user to form a music content representation and a condition matrix includes: Segmenting the music frequency band to form music clips; Each music clip is encoded to form a music content representation; element information of the music clip is obtained, the element information is combined with the music requirement to obtain music combination information, and the music combination information is encoded to form a condition matrix.

3. The controllable generation method of symbolic music as claimed in claim 2, characterized in that: The music discrete diffusion model performs music generation to output a target music score, including: The music content representation and condition matrix of each music clip are concatenated to form a merged representation; Using a preset three-dimensional word embedding space to perform word embedding of the combined representation in a continuous domain to form a combined word embedding vector; Using a preset layer normalization module to perform Markov shift on the merged word embedding vector to obtain a noise initial state of the merged word embedding vector; Performing dynamic noise addition processing on the masked part in the initial noise state through a preset diffusion module to generate a hidden state vector; Applying position word embedding and time step word embedding to the hidden state vector to generate a diffuse hidden vector, and then performing an inverse operation on the diffuse hidden vector through a preset bidirectional encoder network to obtain a target hidden vector; Performing category mapping on the target hidden vector through a preset multi-head output module to obtain a target music content representation matrix; Decoding the target music content representation matrix to generate a target music score frequency band; All target music score frequency bands are concatenated in chronological order to form a target music score.

4. The controllable generation method of symbolic music as claimed in claim 3, characterized in that: Also includes: The target music is generated by loading a music sampling file into the target music score through a preset audio library.

5. The controllable generation method of symbolic music according to claim 1, characterized in that: The pre-processing of the pre-acquired music content and annotation information by a preset characterization module to form an original data set includes: The chords of the music content are used as a timeline, the start time of the notes is automatically quantized in units of 32-notes, the duration of the notes is quantized in units of 16-notes, self-correction is performed, and empty tracks and empty notes are self-deleted to form standard music content, and the standard music content is self-annotated using the annotation information to form a music content set; wherein the music content set contains various music events of the standard music content, and the music events are annotated with music attributes; Combining all music events of a piece of music content into a matrix to form a music content representation, and combining the music content representations of all the music contents to form a music content representation data set; The music content representation dataset is used as the original dataset.

6. The controllable generation method of symbolic music as claimed in claim 5, characterized in that: The masking process is performed on the original data set according to the preset chord condition rule to form an original masked data pair, including: The chord music events represented by the music content in the original data set are displayed, and the non-chord music events represented by the music content are masked; wherein the masked music events are marked with PAD.

7. The controllable generation method of symbolic music according to claim 6, characterized in that: In the process of sending the original masked data pair into a preset music discrete diffusion framework for repeated training to form a music discrete diffusion model, The music discrete framework is trained using original masked data through normalized embedding learning; wherein the music discrete framework includes a diffusion model framework, a decoding model framework and a word embedding module; The norm loss of the diffusion model framework is calculated by a preset BERT model, and the attention mechanism of the diffusion model framework is data-dependent with the word embedding vector involved in the word embedding module; The objective function when training the music discrete diffusion framework is the sum of the loss function of the diffusion model framework, the loss function of the decoding model framework and the loss function of the word embedding module.

8. A controllable generation system of symbolic music, characterized in that: A controllable method for generating symbolic music as claimed in any one of claims 1 to 7 is implemented, wherein the system includes a music discrete diffusion model; wherein: The music discrete diffusion model is generated by learning through repeated training of original masked data pairs using a preset music discrete diffusion framework; wherein obtaining the original masked data pairs includes: preprocessing the pre-acquired music content and annotation information through a preset representation module to form an original data set; performing masking processing on the original data set according to a preset chord condition rule to form an original masked data pair; The music discrete diffusion model is used to receive the music content representation and condition matrix formed by pre-processing the music frequency band and music requirements input by the user, and generate music according to the music content representation and condition matrix to output the target music score.

9. An electronic device, characterized in that: The device includes: a processor, a memory and a system bus; the processor and the memory are connected via the system bus; the memory is used to store one or more programs, and the one or more programs include instructions, which, when executed by the processor, enable the processor to execute the steps in the controllable generation method of symbolic music described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for controllable generation of symbolic music described in any one of claims 1-7 is implemented.

Images