Chord melody generation method and device based on lstm, electronic device and medium

Abstract

The application relates to an LSTM-based chord melody generation method and device, an electronic device and a medium. The method comprises the following steps: performing pre-encoding processing on a first melody segment of a chord melody to be matched, to obtain first fitting parameters; processing the first melody segment by using a trained feature extraction model to generate first front-end label data; generating first label data based on the first front-end label data and the first fitting parameters; encoding the pitch of a note corresponding to the first melody segment to generate a first melody vector; processing the first melody vector and the first label data by using a trained chord prediction model to obtain a plurality of candidate chord segments corresponding to the first melody segment; and selecting a target chord segment corresponding to the first melody segment from the plurality of candidate chord segments based on a preset selection strategy. Through the application, the problem that a chord generated by using a related model according to a melody is unnatural in vision and hearing and has poor adaptation effect with the melody is solved.

Description

Technical Field

[0001] This application relates to the field of AI-assisted music creation technology, and in particular to LSTM-based chord melody generation methods, devices, electronic devices, and media. Background Technology

[0002] In the field of AI-assisted music creation, it generates appropriate chords (the main part of the background music) based on the main melody. It can inspire composition, and the intelligent generation of simple chord sequences also provides more diverse choices and a lower entry threshold for beginners to practice their instruments.

[0003] In related technological fields, deep learning has been applied in speech recognition and sound source separation. However, in the field of music generation, especially in generating chords based on the main melody, there is still a lack of outstanding research and achievements. First, the complexity of music theory is a difficult problem to overcome, involving multiple aspects such as rhythm, pitch, and harmony. To accurately generate chords based on the main melody, a deep understanding and mastery of music theory is required. Second, the aesthetics of music are subjective. Different people may have different feelings and evaluations of the same piece of music. When generating chords, the subjectivity of music needs to be considered, and how to consider this subjectivity presents a certain challenge. Furthermore, the relationship between chords and the main melody is complex. Chords are not simply a stacking of notes; there are rules and logic between them. To accurately generate chords, the rules of chord progression and the overall structure of the music need to be considered.

[0004] The existing chord generation schemes based on melody in related technologies often ignore the rules of chord progressions, and most models are large in size, have many nested models, long training time, and a large number of complex and unintelligent parameters. At the same time, existing models usually only consider the pitch and rhythm of chords, while ignoring other features of chords, such as color and harmony, which makes the generated chord sequences sound unnatural. Furthermore, the models in related technologies do not fully extract the input features of the melody, and the types of chords generated are limited, resulting in poor adaptation to the melody.

[0005] Currently, no effective solution has been proposed for the problem that the generated chords, which use related models to generate chords based on melodies, sound unnatural and do not match the melody well. Summary of the Invention

[0006] This application provides a chord melody generation method, apparatus, electronic device, and medium based on LSTM, to at least solve the problem in related technologies where schemes that use related models to generate chords based on melodies produce chords that sound unnatural and are poorly adapted to the melody.

[0007] In a first aspect, embodiments of this application provide a chord melody generation method based on LSTM, comprising: pre-encoding a first melody segment of the chord melody to be matched to obtain a first fitting parameter, wherein the pre-encoding process is used to characterize the fitting process of the complexity of the melody segment; The first melody fragment is processed using a trained feature extraction model to generate first front-end label data. Based on the first front-end label data and the first fitting parameters, first label data is generated. The first label data is used to characterize the expected target chord fragment to be matched to the first melody fragment. The feature extraction model is a neural network based on a long short-term memory network (LSTM) and trained according to a preset first sample melody fragment and the chord classification corresponding to the first real chord of the first sample melody fragment. The pitch of the notes corresponding to the first melody fragment is encoded to generate a first melody vector. Then, a trained chord prediction model is used to process the first melody vector and the first label data to obtain multiple candidate chord fragments corresponding to the first melody fragment. The chord prediction model is a bidirectional long short-term memory network (BLSTM) and a neural network trained based on the second melody vector, the second real chord, and the second label data corresponding to the second sample melody fragment. The second label data is generated based on the second front-end label data and the second fitting parameters generated by pre-encoding the second sample melody fragment. The second front-end label data is generated by processing the second sample melody fragment using the feature extraction model. Among the multiple candidate chord segments, the target chord segment is selected based on a preset selection strategy, and the target chord segment is used as the chord to match the first melody segment.

[0008] Secondly, embodiments of this application provide an LSTM-based chord melody generation device, comprising: The preprocessing module is used to pre-encode the first melody segment of the chord melody to be matched to obtain the first fitting parameters, wherein the pre-encoding process is used to characterize the fitting process of the complexity of the melody segment; The extraction module is used to process the first melody segment using a trained feature extraction model to generate first front-end label data, and to generate first label data based on the first front-end label data and the first fitting parameters. The first label data is used to characterize the expected target chord segment to be matched to the first melody segment. The feature extraction model is a neural network based on a long short-term memory network (LSTM) and trained according to a preset first sample melody segment and the chord classification corresponding to the first real chord of the first sample melody segment. The prediction module is used to encode the pitch of the notes corresponding to the first melody segment, generate a first melody vector, and use a trained chord prediction model to process the first melody vector and the first label data to obtain multiple candidate chord segments corresponding to the first melody segment. The chord prediction model is a bidirectional long short-term memory network (BLSTM) and a neural network trained based on a preset second melody vector, second real chord, and second label data corresponding to the second sample melody segment. The second label data is generated based on the second front-end label data and the second fitting parameters generated by pre-encoding the second sample melody segment. The second front-end label data is generated by processing the second sample melody segment using the feature extraction model. The processing module is configured to select the target chord segment from among the multiple candidate chord segments based on a preset selection strategy, and use the target chord segment as the chord to match the first melody segment.

[0009] Thirdly, embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the chord melody generation method based on a Long Short-Term Memory (LSTM) network as described in the first aspect.

[0010] Fourthly, embodiments of this application provide a storage medium storing a computer program that, when executed by a processor, implements the steps of the chord melody generation method based on a Long Short-Term Memory (LSTM) network as described in the first aspect above.

[0011] Compared to related technologies, the LSTM-based chord melody generation method, apparatus, electronic device, and medium provided in this application pre-encode a first melody segment of the melody to be matched to obtain a first fitting parameter; process the first melody segment using a trained feature extraction model to generate first front-end label data, and generate first label data based on the first front-end label data and the first fitting parameter; encode the pitch of the notes corresponding to the first melody segment to generate a first melody vector, and process the first melody vector and the first label data using a trained chord prediction model to obtain multiple chords matching the first melody. Candidate chord segments corresponding to the first melody segment; among the multiple candidate chord segments, the target chord segment is selected based on a preset selection strategy, and the target chord segment is used as the chord matching the first melody segment. The corresponding melody segment is pre-encoded, which can preserve the musical properties of the melody segment and improve the accuracy of model training and chord prediction. At the same time, the first melody segment is processed by a feature extraction model and the first label data is generated, which speeds up the training and prediction output of the chord prediction model. This solves the problem in related technologies where the scheme of generating chords based on the melody using related models produces chords that are not natural to the ear and have poor matching effect with the melody.

[0012] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0013] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a hardware structure block diagram of the terminal of the LSTM-based chord melody generation method according to an embodiment of this application; Figure 2 This is a flowchart of an LSTM-based chord melody generation method according to an embodiment of this application; Figure 3 This is a schematic diagram of curve fitting. Figure 4 This is a schematic diagram of a sample melody fragment; Figure 5 This is a structural block diagram of an LSTM-based chord melody generation device according to an embodiment of this application. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application. Furthermore, it is understood that although the efforts made in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, modifications to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0015] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0016] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "a," "an," "an," "the," and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms "comprising," "including," "having," and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. "Multiple stages" used in this application refers to two or more stages. "And / or" describes the relationship between related objects, indicating that three relationships may exist; for example, "A and / or B" can represent: A alone, A and B simultaneously, and B alone. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of objects.

[0017] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. Taking running on a terminal as an example, Figure 1 This is a hardware structure block diagram of the terminal for the LSTM-based chord melody generation method according to an embodiment of this application. Figure 1As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. Optionally, the terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0018] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the LSTM-based chord melody generation method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0019] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0020] This embodiment provides an LSTM-based chord melody generation method running on the aforementioned terminal. Figure 2 This is a flowchart of an LSTM-based chord melody generation method according to an embodiment of this application, as follows: Figure 2 As shown, the process includes the following steps: Step S201: The first melody segment of the chord melody to be matched is pre-encoded to obtain the first fitting parameters, wherein the pre-encoding process is used to characterize the fitting process of the complexity of the melody segment.

[0021] In this embodiment, the first melody fragment is the main melody of a music file in Musical Equipment Digital Interface (MIDI) format, and the chord melody to be matched is the accompaniment adapted to the main melody. In order to obtain the melody features of the first melody fragment so as to generate natural and highly matched chords through subsequent processing, in this embodiment, the first melody fragment is pre-encoded using a fitting method to determine the melody features that characterize the complexity of the first melody fragment, and then obtain the corresponding first fitting parameters. In this embodiment, the first fitting parameters represent the number of fitting operations required to fit the pitch of the notes corresponding to the first melody fragment according to time to achieve a preset goodness of fit, and the number of fitting operations is used as a feature reflecting the complexity of the corresponding first melody fragment.

[0022] Step S202: The first melody fragment is processed using the trained feature extraction model to generate first front-end label data. Based on the first front-end label data and the first fitting parameters, first label data is generated. The first label data is used to characterize the expected target chord fragment to be matched with the first melody fragment. The feature extraction model is a neural network based on a long short-term memory network (LSTM) and trained according to the preset first sample melody fragment and the chord classification corresponding to the first real chord of the first sample melody fragment.

[0023] In this embodiment, after obtaining the first fitting parameters generated by the precoding process, the trained feature extraction model is used to extract preliminary features of the generated chord based on the first fitting parameters. That is, the first front-end label data is used to characterize the expected chord to be generated, for example, a chord with a preset length. Specifically, setting the value corresponding to the first front-end label data to 4 indicates that the chord to be generated is a chord with 4 segments, or it can be a chord of a specific chord type. In this embodiment, 38 chord types are set, and each of the 38 chord types is numbered, with one number corresponding to one chord type. The mapping from specific chord types to integer codes (corresponding to the first front-end label data) is shown in the following table: PAD 0 D#m 15 C 30 pause 1 E 16 Cm 31 A 2 Em 17 Db 32 Am 3 E# 18 Dbm 33 A# 4 E#m 19 Eb 34 A#m 5 F# 20 Ebm 35 B 6 F#m 21 Fb 36 Bm 7 G 22 Fbm 37 B# 8 Gm 23 F 38 B#m 9 G# 24 C# 10 G#m 25 C#m 11 Bb 26 D 12 Bbm 27 Dm 13 Cb 28 D# 14 Cbm 29 In this key, # represents rising, b represents falling, and m represents minor.

[0024] In this embodiment, to further distinguish the nature of chords, a first fitting parameter representing the complexity of the first melody segment is considered to improve the matching degree between the chord classification corresponding to the generated first label data and the first melody segment. For example, if the value corresponding to the first front-end label data is set to 4, the fitting number corresponding to the first fitting parameter is 3, and the weight corresponding to the influence of the first fitting parameter on the first label data is set to 0.1, then the value corresponding to the first label data is 4.3. Therefore, in the subsequent chord prediction process, the predicted chord label has a value of 4.3, which is a chord that can be used as the target chord. It should be understood that the value corresponding to the first label data can represent the matching degree between a certain chord and the first melody segment, or it can be any of the chords in some categories. For example, if the number corresponding to the first label data is 4.3, then the chord corresponding to the first melody segment can be the chords numbered 4 and 5 in the table above.

[0025] In this embodiment, during the training of the feature extraction model, the first real chord corresponding to the first sample melody fragment is one-hot encoded and mapped to the chord category to distinguish the chord classification corresponding to the first real chord. Then, the model is trained based on the corresponding chord classification. When performing feature extraction, the trained feature extraction model can extract the chord classification that initially matches the first chord fragment.

[0026] Step S203: Encode the pitch of the notes corresponding to the first melody fragment to generate a first melody vector. Then, use a trained chord prediction model to process the first melody vector and the first label data to obtain multiple candidate chord fragments corresponding to the first melody fragment. The chord prediction model is a bidirectional long short-term memory network (BLSTM) trained on a neural network based on the second melody vector, the second real chord, and the second label data corresponding to the second sample melody fragment. The second label data is generated based on the second front-end label data and the second fitting parameters generated by pre-encoding the second sample melody fragment. The second front-end label data is generated by processing the second sample melody fragment using a feature extraction model.

[0027] In this embodiment, before using the chord prediction model for chord prediction, the first chord segment needs to be vector-encoded, that is, the pitch of the notes corresponding to the first melody segment is encoded to generate the first melody vector. Then, the first melody vector and the first label data are used as data input to the chord prediction model, and then multiple matching candidate chord segments are output.

[0028] Step S204: Among multiple candidate chord segments, a target chord segment is selected based on a preset selection strategy, and the target chord segment is used as the chord to match the first melody segment.

[0029] Through steps S201 to S204, the first melody fragment of the melody to be matched is pre-encoded to obtain the first fitting parameters; the first melody fragment is processed using a trained feature extraction model to generate the first front-end label data, and the first label data is generated based on the first front-end label data and the first fitting parameters; the pitch of the notes corresponding to the first melody fragment is encoded to generate the first melody vector, and the first melody vector and the first label data are processed using a trained chord prediction model to obtain multiple candidate chord fragments corresponding to the first melody fragment; among the multiple candidate chord fragments, the target chord fragment is selected based on a preset selection strategy, and the target chord fragment is used as the chord to match the first melody fragment. The pre-encoding of the corresponding melody fragment can preserve the musical properties of the melody fragment and improve the accuracy of model training and chord prediction. At the same time, the feature extraction model is used to process the first melody fragment and generate the first label data, which speeds up the training and prediction output of the chord prediction model and solves the problem in related technologies where the generated chords are unnatural in sound and sound and have poor matching effect with the melody when using related models to generate chords based on the melody.

[0030] It should be noted that in this embodiment, the input features are fully extracted and the time factor of the note sequence is considered, so that the matched chords are natural and have a good fit with the first melody segment.

[0031] It should be further explained that the pre-encoding processing based on fitting and the chord classification and matching method based on LSTM are innovative in that they can preserve the properties of MIDI music segments as much as possible while keeping the encoding as concise as possible, thus increasing the effectiveness and training effect of subsequent training. At the same time, although music direction also falls under the category of NLP in the field of natural language processing, most of them are music generation, and there are very few algorithms for music chord matching and simplification. In addition, most music generation relies on large models with millions or even tens of millions of parameters for training. Although both are deep learning, the embodiments of this application have reduced the size of the model as much as possible while ensuring certain effects and speeding up the training speed.

[0032] In some embodiments, the precoding process for the corresponding melody segment includes the following steps: Step 21: Obtain the target melody segment to be processed, wherein the target melody segment includes one of the first melody segment and the second sample melody segment.

[0033] Step 22: Convert the start time and MIDI pitch of each note in the target melody segment to a unified coordinate system to generate two-dimensional coordinate data.

[0034] In this embodiment, the sample melody fragments and the melody fragments to be matched with chords are pre-encoded for the training phase and the chord generation phase, respectively. In this embodiment, the notes on the main melody are drawn on the image according to time and pitch. In this way, the transformation to a unified coordinate system is achieved, and multiple point coordinate data located in the two-dimensional coordinate system are obtained, that is, two-dimensional coordinate data.

[0035] Step 23: Perform polynomial fitting on the two-dimensional coordinate data to obtain the number of fitting iterations that reaches the preset goodness-of-fit threshold. Use the number of fitting iterations as the corresponding target fitting parameter, where the target fitting parameter is one of the first fitting parameter and the second fitting parameter.

[0036] In this embodiment, after plotting the notes of the main melody on the image according to time and pitch, an nth-degree polynomial is used for fitting. Considering that the larger the fitting degree n is, the better the fitting effect will be, in this embodiment, the goodness-of-fit R is used. 2 A minimum threshold was set for it to determine the minimum number of times n required to reach the threshold, which is the target fitting parameter. This target fitting parameter serves as a feature reflecting the complexity of the corresponding melody segment. Figure 3 This is a schematic diagram of curve fitting, for reference. Figure 3 In this embodiment, the start time of each note in the corresponding melody segment is taken as the horizontal axis x, and the MIDI pitch of each note (rests are considered as 0) is taken as the y mark in the two-dimensional coordinate system (the points with stars in the figure are rests and each note). The curve is fitted using a polynomial and the minimum number of times n to achieve the goodness of fit is returned, which is to obtain the target fitting parameters.

[0037] The process involves obtaining the target melody fragment to be processed through the steps described above; converting the start time and MIDI pitch of each note in the target melody fragment to a unified coordinate system to generate two-dimensional coordinate data; performing polynomial fitting on the two-dimensional coordinate data to obtain the number of fits that reaches the preset goodness-of-fit threshold; using the number of fits as the corresponding target fitting parameter; and thus pre-encoding the corresponding melody fragment to obtain melody features that represent the complexity of the melody fragment, providing guidance for the training of the feature extraction model and the extraction of features using the feature extraction model.

[0038] In some embodiments, a target chord segment is selected from multiple candidate chord segments based on a preset selection strategy, which is achieved through the following steps: Step 31: Determine the chord category label corresponding to each candidate chord fragment, wherein the chord category label includes the first probability of the chord category corresponding to the candidate chord fragment.

[0039] Step 32: Select a preset number of first probabilities in descending order of probability value, and use the candidate chord segments corresponding to the selected first probabilities as alternative chord segments.

[0040] Step 33: Use the roulette wheel selection method to select the target chord fragment from the candidate chord fragments.

[0041] In this embodiment, considering that the direction of chords may be uncertain and that chords in popular music have a certain repetition and periodicity, in this embodiment, during the prediction process, the top five candidate chord segments with the highest probability are selected as alternative chord segments and used as the potential output of the chord prediction model. Then, a roulette wheel is used to determine the final target chord segment corresponding to the current first melody segment.

[0042] By determining the chord category label corresponding to each candidate chord fragment in the above steps, wherein the chord category label includes the first probability of the chord category corresponding to the candidate chord fragment; selecting a preset number of first probabilities in descending order of probability value, and using the candidate chord fragments corresponding to the selected first probabilities as candidate chord fragments; and using the roulette wheel selection method to select the target chord fragment from the candidate chord fragments, the target chord fragment with high matching degree and natural audiovisual sound is determined.

[0043] In some embodiments, the training steps of the feature extraction model include the following steps: Step 41: Obtain the first sample melody fragment and the first actual score chord from the first MIDI sample music file.

[0044] In this embodiment, 1824 segments were selected from 14612 MIDI music segments. These segments were mainly composed of block chord accompaniment textures, and preliminary chord and melody features were extracted.

[0045] Step 42: Encode the pitch of the notes corresponding to the first sample melody fragment to generate the first sample melody vector. After selecting the first standard melody vector with the longest vector length from the first sample melody vectors, perform padding and word embedding on all the first sample melody vectors to generate the second standard melody vector. The vector length of the second standard melody vector is equal to the vector length of the first standard melody vector.

[0046] In this embodiment, when encoding the pitch of the notes corresponding to the first sample melody fragment, the pitch is directly converted into a corresponding digital vector, for example: (Refer to...) Figure 4 The sample melody fragment shown is encoded in pitch to generate the following first sample melody vector [0,60,60,60,62,64].

[0047] In this embodiment, since the lengths of the generated first melody vectors vary, in order to complete the training of the LSTM, the length of the longest first melody vector is first calculated, and then padding is used to make the length of each first melody vector equal to the length of the longest first melody vector. Then, word embedding is performed on each first melody vector to generate a second standard melody vector with the same length.

[0048] Step 43: Determine the first one-hot vector encoding parameters corresponding to the first real chord, wherein the first one-hot vector encoding parameters are used to characterize the chord type and chord length corresponding to the first real chord.

[0049] In this embodiment, for the chord of the sample, the stored chord mapping table is used to determine the corresponding one-hot vector encoding parameters, and then the chord classification of the real chord corresponding to the sample chord segment is determined, such as the chord type and chord length of the real chord.

[0050] Step 44: Train the LSTM using the second standard melody vector and the first one-hot vector encoding parameters until convergence, generating a feature extraction model.

[0051] In this embodiment, the AdamW optimizer and cosine annealing algorithm are used to dynamically adjust the learning rate.

[0052] In some embodiments, determining the first one-hot vector encoding parameters corresponding to the first real spectral chord in step 43 includes the following steps: Step 431: Obtain the one-hot vector parameter table, which includes the third real spectral chord, the second one-hot vector encoding parameter, and the correspondence information between the third real spectral chord and the second one-hot vector encoding parameter.

[0053] Step 432: In the one-hot vector parameter table, find the second one-hot vector encoding parameter corresponding to the third real chord that is the same as the first real chord, and obtain the first one-hot vector encoding parameter.

[0054] In some embodiments, the first label data is generated based on the first front-end label data and the first fitting parameters, through the following steps: Step 51: Obtain the first front-end label data and the first fitting parameters.

[0055] Step 52, obtain the first label data L according to the following formula: L=L0+ln, where L0 represents the first front-end label data, l represents the weight of the fitting parameters, and n represents the first fitting parameters.

[0056] In this embodiment, a dictionary is set up for the chords themselves to store the mapping relationship between different chords. At the same time, the fitting parameters obtained by precoding are also used as the properties of the chords themselves. The first label data obtained by the above formula is used as the basis for the one-hot vector encoding of the chord part, so as to better reflect the characteristics of different chords.

[0057] In some embodiments, the training steps of the chord prediction model include: Step 61: Extract the second sample melody fragment and the second real score chord from the second MIDI sample music file, and use the feature extraction model to extract the corresponding second front-end label data from the second sample melody fragment.

[0058] Step 62: Pre-encode the second sample melody segment to generate the second fitting parameters, and generate the second label data based on the second front-end label data and the second fitting parameters.

[0059] Step 63: Encode the pitch of the notes corresponding to the second sample melody fragment to generate a second melody vector. Input the second melody vector and the second label data as the training set and the second real score chord as the validation set into the BLSTM network for training until convergence, and obtain the chord prediction model. During the training process, the cosine annealing algorithm is used to adjust the learning rate.

[0060] This embodiment also provides an LSTM-based chord melody generation device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, terms such as "module," "unit," and "subunit" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0061] Figure 5 This is a structural block diagram of an LSTM-based chord melody generation device according to an embodiment of this application, as shown below. Figure 5 As shown, the device includes a preprocessing module 51, an extraction module 52, a prediction module 53, and a processing module 54, wherein... The preprocessing module 51 is used to pre-encode the first melody segment of the chord melody to be matched to obtain the first fitting parameters, wherein the pre-encoding process is used to characterize the fitting process of the complexity of the melody segment.

[0062] The extraction module 52, coupled to the preprocessing module 51, is used to process the first melody fragment using a trained feature extraction model, generate first front-end label data, and generate first label data based on the first front-end label data and the first fitting parameters. The first label data is used to characterize the expected target chord fragment to be matched to the first melody fragment. The feature extraction model is a neural network based on a long short-term memory network (LSTM) and trained according to a preset first sample melody fragment and the chord classification corresponding to the first real chord of the first sample melody fragment.

[0063] The prediction module 53, coupled to the preprocessing module 51 and the extraction module 52, is used to encode the pitch of the notes corresponding to the first melody segment, generate the first melody vector, and use a trained chord prediction model to process the first melody vector and the first label data to obtain multiple candidate chord segments corresponding to the first melody segment. The chord prediction model is a bidirectional long short-term memory network (BLSTM) and a neural network trained based on the second melody vector, the second real chord, and the second label data corresponding to the second sample melody segment. The second label data is generated based on the second front-end label data and the second fitting parameters generated by pre-encoding the second sample melody segment. The second front-end label data is generated by processing the second sample melody segment using a feature extraction model.

[0064] The processing module 54, coupled to the prediction module 53, is used to select a target chord fragment from multiple candidate chord fragments based on a preset selection strategy, and to use the target chord fragment as the chord to match the first melody fragment.

[0065] The LSTM-based chord melody generation device of this application pre-encodes a first melody segment of the melody to be matched to obtain first fitting parameters; processes the first melody segment using a trained feature extraction model to generate first front-end label data, and generates first label data based on the first front-end label data and the first fitting parameters; encodes the pitch of the notes corresponding to the first melody segment to generate a first melody vector, and processes the first melody vector and the first label data using a trained chord prediction model to obtain multiple candidate chord segments corresponding to the first melody segment; selects a target chord segment from the multiple candidate chord segments based on a preset selection strategy, and uses the target chord segment as the chord to match the first melody segment. By pre-encoding the corresponding melody segment, the musical properties of the melody segment can be preserved, improving the accuracy of model training and chord prediction. At the same time, by using a feature extraction model to process the first melody segment and generate first label data, the training speed and prediction output speed of the chord prediction model are accelerated, solving the problem in related technologies where chords generated by using related models to generate chords based on melodies are unnatural in sound and sound and have poor adaptation to the melody.

[0066] In some embodiments, the preprocessing module 51 is further configured to: acquire a target melody segment to be processed, wherein the target melody segment includes one of a first melody segment and a second sample melody segment; convert the start time and MIDI pitch of each note in the target melody segment to a unified coordinate system to generate two-dimensional coordinate data; perform polynomial fitting processing on the two-dimensional coordinate data to obtain the number of fitting iterations that reaches a preset goodness-of-fit threshold, and use the number of fitting iterations as the corresponding target fitting parameter, wherein the target fitting parameter is one of a first fitting parameter and a second fitting parameter.

[0067] In some embodiments, the processing module 54 further includes: The first determining unit is used to determine the chord category label corresponding to each candidate chord fragment, wherein the chord category label includes a first probability of the chord category corresponding to the candidate chord fragment.

[0068] The first selection unit, coupled to the first determination unit, is used to select a preset number of first probabilities in descending order of probability value, and to use the candidate chord segments corresponding to the selected first probabilities as alternative chord segments.

[0069] The first selection unit, coupled to the first selection unit, is used to select the target chord segment from the candidate chord segments using the roulette wheel selection method.

[0070] In some embodiments, the LSTM-based chord melody generation device is used to obtain a first sample melody fragment and a first real-score chord from a first MIDI sample music file; encode the pitch of the notes corresponding to the first sample melody fragment to generate a first sample melody vector; after selecting the first standard melody vector with the longest vector length from the first sample melody vectors, perform padding and word embedding processing on all first sample melody vectors to generate a second standard melody vector, wherein the vector length of the second standard melody vector is equal to the vector length of the first standard melody vector; determine the first one-hot vector encoding parameters corresponding to the first real-score chord, wherein the first one-hot vector encoding parameters are used to characterize the chord type and chord length corresponding to the first real-score chord; and train the LSTM using the second standard melody vector and the first one-hot vector encoding parameters until convergence to generate a feature extraction model.

[0071] In some embodiments, the LSTM-based chord melody generation device is further used to obtain a hot vector parameter table, wherein the hot vector parameter table includes a third real chord, a second hot vector encoding parameter, and the correspondence information between the third real chord and the second hot vector encoding parameter; in the hot vector parameter table, the second hot vector encoding parameter corresponding to the third real chord that is the same as the first real chord is found to obtain the first hot vector encoding parameter.

[0072] In some embodiments, the extraction module 52 is further configured to: acquire first front-end label data and first fitting parameters;

[0073] The first label data L is obtained by the following formula: L = L0 + ln, where L0 represents the first front-end label data, l represents the weight of the fitting parameters, and n represents the first fitting parameters.

[0074] In some embodiments, the LSTM-based chord melody generation device is used to extract a second sample melody fragment and a second real-score chord from a second MIDI sample music file, and to extract corresponding second front-end label data from the second sample melody fragment using a feature extraction model; to pre-encode the second sample melody fragment to generate second fitting parameters, and to generate second label data based on the second front-end label data and the second fitting parameters; to encode the pitch of the notes corresponding to the second sample melody fragment to generate a second melody vector, and to input the second melody vector and the second label data as a training set and the second real-score chord as a validation set into a BLSTM network for training until convergence, thereby obtaining a chord prediction model. During the training process, a cosine annealing algorithm is used to adjust the learning rate.

[0075] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0076] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0077] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program: S1, pre-encode the first melody segment of the chord melody to be matched to obtain the first fitting parameters, wherein the pre-encoding process is used to characterize the fitting process of the complexity of the melody segment.

[0078] S2, the first melody fragment is processed using the trained feature extraction model to generate the first front-end label data, and the first label data is generated based on the first front-end label data and the first fitting parameters. The first label data is used to characterize the expected target chord fragment to be matched to the first melody fragment. The feature extraction model is a neural network based on the Long Short-Term Memory Network (LSTM) and trained according to the preset first sample melody fragment and the chord classification corresponding to the first real chord of the first sample melody fragment.

[0079] S3, the pitch of the notes corresponding to the first melody fragment is encoded to generate the first melody vector, and the first melody vector and the first label data are processed using the trained chord prediction model to obtain multiple candidate chord fragments corresponding to the first melody fragment. The chord prediction model is a bidirectional long short-term memory network BLSTM, and is a neural network trained based on the second melody vector, the second real chord, and the second label data corresponding to the second sample melody fragment. The second label data is generated based on the second front-end label data and the second fitting parameters generated by pre-encoding the second sample melody fragment. The second front-end label data is generated by processing the second sample melody fragment using a feature extraction model.

[0080] S4: Among multiple candidate chord segments, select the target chord segment based on a preset selection strategy, and use the target chord segment as the chord to match the first melody segment.

[0081] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0082] In addition, in conjunction with the LSTM-based chord melody generation method in the above embodiments, this application embodiment can provide a storage medium storing a computer program; when the computer program is executed by a processor, it implements any of the LSTM-based chord melody generation methods in the above embodiments.

[0083] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0084] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A chord melody generation method based on Long Short-Term Memory (LSTM) network, characterized in that, include: The first melodic segment of the chord melody to be matched is pre-encoded to obtain the first fitting parameters, wherein the pre-encoding process is used to characterize the fitting process of the complexity of the melodic segment; The first melody fragment is processed using a trained feature extraction model to generate first front-end label data. Based on the first front-end label data and the first fitting parameters, first label data is generated. The first label data is used to characterize the expected target chord fragment to be matched to the first melody fragment. The feature extraction model is a neural network based on a long short-term memory network (LSTM) and trained according to a preset first sample melody fragment and the chord classification corresponding to the first real chord of the first sample melody fragment. The pitch of the notes corresponding to the first melody fragment is encoded to generate a first melody vector. Then, a trained chord prediction model is used to process the first melody vector and the first label data to obtain multiple candidate chord fragments corresponding to the first melody fragment. The chord prediction model is a bidirectional long short-term memory network (BLSTM) trained on a neural network based on a preset second melody vector, second real chord, and second label data corresponding to a second sample melody fragment. The second label data is generated based on second front-end label data and second fitting parameters generated by pre-encoding the second sample melody fragment. The second front-end label data is generated by processing the second sample melody fragment using the feature extraction model. Among the multiple candidate chord segments, the target chord segment is selected based on a preset selection strategy, and the target chord segment is used as the chord to match the first melody segment.

2. The method according to claim 1, characterized in that, The pre-coding process for the corresponding melody segment includes: Obtain the target melody segment to be processed, wherein the target melody segment includes one of the first melody segment and the second sample melody segment; The start time and MIDI pitch of each note in the target melody segment are converted to a unified coordinate system to generate two-dimensional coordinate data; The two-dimensional coordinate data is subjected to polynomial fitting to obtain the number of fitting iterations that reaches a preset goodness-of-fit threshold. The number of fitting iterations is used as the corresponding target fitting parameter, wherein the target fitting parameter is one of the first fitting parameter and the second fitting parameter.

3. The method according to claim 1, characterized in that, Among the multiple candidate chord segments, the target chord segment is selected based on a preset selection strategy, including: Determine the chord category label corresponding to each candidate chord segment, wherein the chord category label includes a first probability of the chord category corresponding to the candidate chord segment; Select a preset number of the first probabilities in descending order of probability value, and use the candidate chord segments corresponding to the selected first probabilities as alternative chord segments. The target chord segment is selected from the candidate chord segments using a roulette wheel selection method.

4. The method according to claim 1, characterized in that, The training steps of the feature extraction model include: Obtain the first sample melody fragment and the first actual score chord from the first MIDI sample music file; The pitch of the notes corresponding to the first sample melody segment is encoded to generate a first sample melody vector. After selecting the first standard melody vector with the longest vector length from the first sample melody vectors, all first sample melody vectors are padded with zeros and word embedded to generate a second standard melody vector. The vector length of the second standard melody vector is equal to the vector length of the first standard melody vector. Determine the first one-hot vector encoding parameters corresponding to the first real chord, wherein the first one-hot vector encoding parameters are used to characterize the chord type and chord length corresponding to the first real chord; The LSTM is trained using the second standard melody vector and the first one-hot vector encoding parameters until convergence, thereby generating the feature extraction model.

5. The method according to claim 4, characterized in that, Determining the first one-hot vector encoding parameters corresponding to the first real spectral chord includes: Obtain the one-hot vector parameter table, wherein the one-hot vector parameter table includes the third real spectral chord, the second one-hot vector encoding parameter, and the correspondence information between the third real spectral chord and the second one-hot vector encoding parameter; In the one-hot vector parameter table, the second one-hot vector encoding parameter corresponding to the third real chord that is the same as the first real chord is found, and the first one-hot vector encoding parameter is obtained.

6. The method according to claim 4, characterized in that, Based on the first front-end label data and the first fitting parameters, first label data is generated, including: Obtain the first front-end label data and the first fitting parameters; The first tag data L is obtained using the following formula: L = L0 + ln Where L0 represents the first front-end label data, l represents the weight of the fitting parameters, and n represents the first fitting parameters.

7. The method according to claim 4, characterized in that, The training steps of the chord prediction model include: Extract the second sample melody fragment and the second actual score chord from the second MIDI sample music file, and use the feature extraction model to extract the corresponding second front-end tag data from the second sample melody fragment; The second sample melody segment is pre-encoded to generate the second fitting parameters, and the second label data is generated based on the second front-end label data and the second fitting parameters; The pitch of the notes corresponding to the second sample melody fragment is encoded to generate a second melody vector. The second melody vector and the second label data are used as the training set, and the second real score chord is used as the validation set. The BLSTM network is then trained until convergence, resulting in the chord prediction model. During the training process, the learning rate is adjusted using a cosine annealing algorithm.

8. A chord melody generation device based on LSTM, characterized in that, include: The preprocessing module is used to pre-encode the first melody segment of the chord melody to be matched to obtain the first fitting parameters, wherein the pre-encoding process is used to characterize the fitting process of the complexity of the melody segment. The extraction module is used to process the first melody segment using a trained feature extraction model to generate first front-end label data, and to generate first label data based on the first front-end label data and the first fitting parameters. The first label data is used to characterize the expected target chord segment to be matched to the first melody segment. The feature extraction model is a neural network based on a long short-term memory network (LSTM) and trained according to a preset first sample melody segment and the chord classification corresponding to the first real chord of the first sample melody segment. The prediction module is used to encode the pitch of the notes corresponding to the first melody segment, generate a first melody vector, and use a trained chord prediction model to process the first melody vector and the first label data to obtain multiple candidate chord segments corresponding to the first melody segment. The chord prediction model is a bidirectional long short-term memory network (BLSTM) and a neural network trained based on a preset second melody vector, second real chord, and second label data corresponding to the second sample melody segment. The second label data is generated based on the second front-end label data and the second fitting parameters generated by pre-encoding the second sample melody segment. The second front-end label data is generated by processing the second sample melody segment using the feature extraction model. The processing module is configured to select the target chord segment from among the multiple candidate chord segments based on a preset selection strategy, and use the target chord segment as the chord to match the first melody segment.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the steps of the chord melody generation method based on a Long Short-Term Memory (LSTM) network as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the chord melody generation method based on Long Short-Term Memory (LSTM) network as described in any one of claims 1 to 7.

Images