Training method, device, medium and computing equipment for song chord arrangement model

Abstract

The embodiments of the present disclosure provide a training method, apparatus, medium and computing device for a song chord arrangement model. The method comprises: extracting audio feature information from an audio file of a sample song, wherein the audio feature information comprises song chords and a sequence of notes corresponding to human voice; generating a note matrix corresponding to the note sequence according to the audio feature information; and using the note matrix as a training sample to perform supervised training on the song chord arrangement model, wherein the extracted song chords are used as sample labels of the training samples. The method enables composers to use the song chord arrangement model to achieve automatic arrangement of song chords. Not only can it significantly reduce the time required for chord arrangement, thereby improving the arrangement efficiency of composers and reducing labor costs; but even ordinary users with a weak foundation in music theory can achieve chord arrangement, thereby contributing to the realization of more diverse forms of music services.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of audio processing technology. More specifically, the embodiments of the present disclosure relate to a training method, apparatus, medium and computing device for a song chord arrangement model. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] The information that constitutes music is extremely complex, usually including melody and accompaniment, and chords are an important part of accompaniment. Chords refer to a set of pitch logic composed of several notes according to a certain interval relationship. It is of great significance to strengthen the emotional color of the melody and enhance the expressiveness of the music work. Chord arrangement is usually the core work in the accompaniment arrangement process and an important part of the composition work.

[0004] In current composing work, chord arrangement is usually done manually by composers, that is, manually writing music scores to complete the chord arrangement work. Summary of the invention

[0005] However, in order to ensure the quality of chord arrangement, manual arrangement usually has high requirements for the arranger, such as requiring a solid foundation in music theory, etc. Moreover, the process of manual chord arrangement is usually cumbersome and time-consuming, so the arrangement efficiency is generally low and the labor cost is high.

[0006] Therefore, there is a great need for an improved training method for song chord arrangement model to improve the efficiency of chord arrangement.

[0007] In the context of this specification, embodiments of the present disclosure are intended to provide a method, apparatus, medium, and computing device for training a song chord arrangement model.

[0008] In a first aspect of the embodiments of the present disclosure, a method for training a song chord arrangement model is provided, the method comprising:

[0009] Extracting audio feature information from an audio file of a sample song, wherein the audio feature information includes song chords and a note sequence corresponding to a human voice;

[0010] generating a note matrix corresponding to the note sequence according to the audio feature information;

[0011] The song chord arrangement model is supervisedly trained using the note matrix as a training sample, wherein the extracted song chords are used as sample labels of the training samples.

[0012] Optionally, the audio feature information further includes duration information of each vocal note in the note sequence, and the method further includes:

[0013] Taking a preset multiple of diaeresis as a basic unit, determining a beat point sequence including a plurality of reference beat points in a time interval corresponding to the note sequence;

[0014] The reference note beat points corresponding to the respective vocal notes in the note sequence in the beat point sequence are determined according to the duration information of the respective vocal notes, and the reference note beat points corresponding to any vocal note are used to determine the matrix element corresponding to the vocal note in the note matrix.

[0015] Optionally, it also includes:

[0016] The note beat points corresponding to the preset moments of each vocal note in the note sequence are adjusted to the corresponding nearest reference note beat points; wherein the nearest reference note beat point corresponding to any note beat is the reference beat point closest to the note beat point in the beat point sequence, and the adjusted reference note beat points corresponding to each vocal note are used to generate the matrix elements corresponding to the vocal note in the note matrix.

[0017] Optionally, the preset multiple of the quaver notes is thirty-second notes.

[0018] Optionally, the audio feature information further includes pitch information of each vocal note in the note sequence, and generating a note matrix corresponding to the note sequence according to the audio feature information includes:

[0019] According to the pitch information corresponding to each vocal note and the determined reference note beat point, the matrix element corresponding to the vocal note in the note matrix is ​​determined to generate the note matrix.

[0020] Optionally, the note matrix is ​​an m×n matrix, wherein the number of rows m is the number of notes in the note sequence used to generate the note matrix, and the number of columns n is the number of beats of the beat point sequence divided according to preset multiples of the reference beat point; the step of determining the matrix element corresponding to the vocal note in the note matrix according to the pitch information corresponding to each vocal note and the determined reference note beat point to generate the note matrix comprises:

[0021] Determine the row corresponding to each vocal note in the note matrix according to the pitch information corresponding to each vocal note; and

[0022] Determine the columns corresponding to the vocal notes in the note matrix according to the reference note beat points corresponding to the vocal notes in the note sequence;

[0023] Determine the non-zero elements in the note matrix according to the row and column positions corresponding to each individual voice note, and set the non-zero elements to a preset value to generate the note matrix.

[0024] Optionally, the audio feature information further includes the song mode of the sample song and the pitch information of each individual voice note in the note sequence, and the pitch information matches the song mode. The method further includes:

[0025] When the song mode is different from the standard mode, adjust the pitch information to the pitch information that matches the standard mode, and the adjusted pitch information is used to generate the note matrix.

[0026] Optionally, the generating a note matrix corresponding to the note sequence according to the audio feature information includes:

[0027] Determine the beats in the sample song, and sequentially divide the note sequence into multiple note segments according to a preset number of beats. Any note segment corresponds to the audio segment feature information corresponding to the corresponding beat interval in the audio feature information;

[0028] Generate note sub-matrices corresponding to each note segment according to the note segment feature information corresponding to each note segment in the note sequence, and the chord segment corresponding to any note segment in the extracted song chords is used as the sample label of the note sub-matrix corresponding to the note segment.

[0029] Optionally, the preset number of beats is: the number of beats per measure in the song beats of the sample song.

[0030] Optionally, the audio feature information further includes the song mode of the sample song; calculate the reward Reward corresponding to the song chord arrangement result O output by the song chord arrangement model for any note segment through the following formula (1) or (2):

[0031]

[0032]

[0033] Wherein, a, b, and c are all positive numbers, and b < c. DL is the chord segment corresponding to any note segment in the extracted song chords, and key C is the chord set composed of alternative chords corresponding to the song mode.

[0034] Optionally, the song chord arrangement model is built based on a reinforcement learning model. The agent in the song chord arrangement model includes a first network composed of multiple one-dimensional convolutions and a second network composed of multiple fully connected layers.

[0035] In a second aspect of the embodiments of the present disclosure, a method for arranging song chords is provided, the method comprising:

[0036] Acquire an audio file of a song to be arranged, and extract a note sequence corresponding to a human voice in the song to be arranged from the audio file;

[0037] generating a note matrix corresponding to the note sequence according to the note sequence;

[0038] The note matrix is ​​input into a pre-trained song chord arrangement model, and the song chords output by the song chord arrangement model after arranging the song to be arranged based on the note matrix are received.

[0039] Optionally, generating a note matrix corresponding to the note sequence according to the note sequence includes:

[0040] Determining the beats in the song to be arranged, and dividing the note sequence into a plurality of note segments in sequence according to a preset number of beats;

[0041] Determine the pitch information and duration information of each vocal note of the song to be arranged according to the note sequence;

[0042] The note sub-matrices corresponding to the respective note segments are generated respectively according to the pitch information and duration information of the respective vocal notes.

[0043] Optionally, the step of inputting the note matrix into a pre-trained song chord arrangement model, and receiving the song chords output by the song chord arrangement model after arranging the song to be arranged based on the note matrix, comprises:

[0044] Input a sub-matrix set consisting of each note sub-matrix corresponding to each note segment into the song chord arrangement model as an episode;

[0045] The song chords output by the song chord arrangement model are received, wherein the song chords include chord segments corresponding to the respective note segments.

[0046] Optionally, the song chord arrangement model is obtained by pre-training with sample songs, wherein the note matrix of the sample song is used as a training sample, and the song chords of the sample song are used as sample labels of the training sample.

[0047] Optionally, the song and chord arrangement model is built based on a reinforcement learning model, and the intelligent agent Agent in the song and chord arrangement model includes a first network composed of multiple layers of one-dimensional convolution and a second network composed of multiple fully connected layers.

[0048] Optionally, the song chord arrangement model arranges the song to be arranged based on the note matrix, including:

[0049] The first network extracts feature information of the song to be arranged based on the note matrix;

[0050] The second network determines the chords corresponding to the respective vocal notes in the note sequence according to the feature information.

[0051] Optionally, the song chord arrangement model is pre-trained through sample songs, the song mode of the sample songs corresponds to multiple alternative chords, and the size of the reward corresponding to the chord arrangement result output by the song chord arrangement model for the sample song is positively correlated with the degree of matching between the chord arrangement result and the alternative chords.

[0052] Optionally, the audio file includes:

[0053] A vocal file containing the vocals in the song to be arranged; or,

[0054] A MIDI file generated based on a vocal file containing the vocals in the song to be arranged.

[0055] Optionally, the song chords include:

[0056] The independent chord file corresponding to the song to be arranged;

[0057] The independent chord file corresponding to the song to be arranged is merged into the audio file to obtain a complete song audio file.

[0058] Optionally, the audio file is provided by a user, and the song chords are used to be displayed to the user.

[0059] In a third aspect of the embodiments of the present disclosure, a training device for a song chord arrangement model is provided, the device comprising:

[0060] An information extraction module, used to extract audio feature information from the audio file of the sample song, wherein the audio feature information includes song chords and a note sequence corresponding to human voice;

[0061] A matrix generation module, used for generating a note matrix corresponding to the note sequence according to the audio feature information;

[0062] The model training module is used to perform supervised training on the song chord arrangement model by using the note matrix as a training sample, wherein the extracted song chords are used as sample labels of the training samples.

[0063] Optionally, the audio feature information further includes duration information of each vocal note in the note sequence, and the device further includes:

[0064] A sequence determination module, for determining a beat point sequence including a plurality of reference beat points in a time interval corresponding to the note sequence, taking a preset multiple of diaeresis notes as a basic unit;

[0065] A beat point determination module is used to determine the reference note beat points corresponding to each vocal note in the note sequence in the beat point sequence according to the duration information of each vocal note, and the reference note beat point corresponding to any vocal note is used to determine the matrix element corresponding to the vocal note in the note matrix.

[0066] Optionally, it also includes:

[0067] A beat point adjustment module is used to adjust the note beat points corresponding to the preset moments of each vocal note in the note sequence to the corresponding nearest reference note beat points; wherein the nearest reference note beat point corresponding to any note beat is the reference beat point closest to the note beat point in the beat point sequence, and the adjusted reference note beat points corresponding to each vocal note are used to generate the matrix elements corresponding to the vocal note in the note matrix.

[0068] Optionally, the preset multiple of the quaver notes is thirty-second notes.

[0069] Optionally, the audio feature information further includes pitch information of each vocal note in the note sequence, and the matrix generation module is further used for:

[0070] According to the pitch information corresponding to each vocal note and the determined reference note beat point, the matrix element corresponding to the vocal note in the note matrix is ​​determined to generate the note matrix.

[0071] Optionally, the note matrix is ​​an m×n matrix, wherein the number of rows m is the number of notes in the note sequence used to generate the note matrix, and the number of columns n is the number of beats of the beat point sequence divided according to preset multiples of the reference beat point; the matrix generation module is further used to:

[0072] Determine the row corresponding to each vocal note in the note matrix according to the pitch information corresponding to each vocal note; and

[0073] Determine the columns corresponding to the vocal notes in the note matrix according to the reference note beat points corresponding to the vocal notes in the note sequence;

[0074] Determine the non-zero elements in the note matrix according to the row and column positions corresponding to each personal voice note, and set the non-zero elements to a preset value to generate the note matrix.

[0075] Optionally, the audio feature information further includes the key of the sample song and the pitch information of each personal voice note in the note sequence, the pitch information being matched to the key of the song, and the apparatus further includes:

[0076] A key adjustment module, configured to, when the key of the song is different from the standard key, adjust the pitch information to the pitch information that matches the standard key, and the adjusted pitch information is used to generate the note matrix.

[0077] Optionally, the matrix generation module is further configured to:

[0078] Determine the beats in the sample song, and sequentially divide the note sequence into multiple note segments according to a preset number of beats, and any note segment corresponds to the audio segment feature information corresponding to the corresponding beat interval in the audio feature information;

[0079] Generate respective note sub-matrices corresponding to the respective note segments according to the note segment feature information corresponding to the respective note segments in the note sequence, and the chord segment corresponding to any note segment in the extracted song chords is used as the sample label of the note sub-matrix corresponding to the note segment.

[0080] Optionally, the preset number of beats is: the number of beats per measure in the song beats of the sample song.

[0081] Optionally, the audio feature information further includes the key of the sample song;

[0082] The apparatus further includes a reward calculation module, configured to calculate the reward Reward corresponding to the song chord arrangement result O output by the song chord arrangement model for any note segment through the following formula (1) or (2):

[0083]

[0084]

[0085] where a, b, and c are all positive numbers, and b < c, DL is the chord segment corresponding to the any note segment in the extracted song chords, and key C is the chord set composed of the alternative chords corresponding to the key of the song.

[0086] Optionally, the song and chord arrangement model is built based on a reinforcement learning model, and the intelligent agent Agent in the song and chord arrangement model includes a first network composed of multiple layers of one-dimensional convolution and a second network composed of multiple fully connected layers.

[0087] In a fourth aspect of the embodiments of the present disclosure, a device for arranging song chords is provided, the device comprising:

[0088] A file acquisition module, used to acquire an audio file of a song to be arranged, and extract a note sequence corresponding to the human voice in the song to be arranged from the audio file;

[0089] A matrix generation module, used for generating a note matrix corresponding to the note sequence according to the note sequence;

[0090] The chord arrangement module is used to input the note matrix into a pre-trained song chord arrangement model, and receive the song chords output by the song chord arrangement model after arranging the song to be arranged based on the note matrix.

[0091] Optionally, the matrix generation module is further used for:

[0092] Determining the beats in the song to be arranged, and dividing the note sequence into a plurality of note segments in sequence according to a preset number of beats;

[0093] Determine the pitch information and duration information of each vocal note of the song to be arranged according to the note sequence;

[0094] The note sub-matrices corresponding to the respective note segments are generated respectively according to the pitch information and duration information of the respective vocal notes.

[0095] Optionally, the chord arrangement module is further used for:

[0096] Input a sub-matrix set consisting of each note sub-matrix corresponding to each note segment into the song chord arrangement model as an episode;

[0097] The song chords output by the song chord arrangement model are received, wherein the song chords include chord segments corresponding to the respective note segments.

[0098] Optionally, the song chord arrangement model is obtained by pre-training with sample songs, wherein the note matrix of the sample song is used as a training sample, and the song chords of the sample song are used as sample labels of the training sample.

[0099] Optionally, the song and chord arrangement model is built based on a reinforcement learning model, and the intelligent agent Agent in the song and chord arrangement model includes a first network composed of multiple layers of one-dimensional convolution and a second network composed of multiple fully connected layers.

[0100] Optionally, the chord arrangement module is further used for:

[0101] The first network extracts feature information of the song to be arranged based on the note matrix;

[0102] The second network determines the chords corresponding to the respective vocal notes in the note sequence according to the feature information.

[0103] Optionally, the song chord arrangement model is pre-trained through sample songs, the song mode of the sample songs corresponds to multiple alternative chords, and the size of the reward corresponding to the chord arrangement result output by the song chord arrangement model for the sample song is positively correlated with the degree of matching between the chord arrangement result and the alternative chords.

[0104] Optionally, the audio file includes:

[0105] A vocal file containing the vocals in the song to be arranged; or,

[0106] A MIDI file generated based on a vocal file containing the vocals in the song to be arranged.

[0107] Optionally, the song chords include:

[0108] The independent chord file corresponding to the song to be arranged;

[0109] The independent chord file corresponding to the song to be arranged is merged into the audio file to obtain a complete song audio file.

[0110] Optionally, the audio file is provided by a user, and the song chords are used to be displayed to the user.

[0111] In a fifth aspect of the embodiments of the present disclosure, a medium is provided on which a computer program is stored. When the computer program is executed by a processor, the method for training a song chord arrangement model as described in any one of the embodiments in the first aspect above is implemented.

[0112] In a sixth aspect of the embodiments of the present disclosure, there is provided a computing device, including:

[0113] processor;

[0114] a memory for storing processor-executable instructions;

[0115] The processor implements the training method of the song chord arrangement model described in any embodiment of the first aspect by running the executable instructions.

[0116] According to the training method of the song and chord arrangement model of the embodiment of the present disclosure, the model training device can extract audio feature information including song chords and note sequences corresponding to human voices from the audio files of the sample songs, and then generate a note matrix corresponding to the note sequence based on the audio feature information, and then use the matrix as a training sample to perform supervised training on the song and chord arrangement model, wherein the extracted song chords mentioned above are used as sample labels of the training samples.

[0117] By adopting the above training method, a song chord arrangement model can be provided to composers, so that composers can use the model to realize automatic arrangement of song chords. This allows composers to focus more on the music itself rather than a lot of repetitive work. Automatic arrangement of song chords through this model can not only greatly reduce the time required for chord arrangement, but also improve the arrangement efficiency of composers and reduce labor costs. Moreover, even ordinary users with weak music theory foundation can realize chord arrangement, which helps to realize more diversified forms of music services. BRIEF DESCRIPTION OF THE DRAWINGS

[0118] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0119] Figure 1 A schematic diagram of a song arrangement system in the related art is schematically shown;

[0120] Figure 2 A flowchart of a method for training a song chord arrangement model according to an embodiment of the present disclosure is schematically shown;

[0121] Figure 3 A schematic diagram schematically shows a song feature information extraction process according to an embodiment of the present disclosure;

[0122] Figure 4 A schematic diagram schematically shows a song tune adjustment process according to an embodiment of the present disclosure;

[0123] Figure 5 A schematic diagram schematically shows a beat point adjustment process according to an embodiment of the present disclosure;

[0124] Figure 6 A schematic diagram schematically shows a MIDI sequence number according to an embodiment of the present disclosure;

[0125] Figure 7 A schematic diagram of a sparse matrix according to an embodiment of the present disclosure is schematically shown;

[0126] Figure 8 A schematic diagram of a note matrix according to an embodiment of the present disclosure is schematically shown;

[0127] Fig. 9 The structure diagram of a song chord arrangement model based on a reinforcement learning model according to an embodiment of the present disclosure is schematically shown;

[0128] Fig.10 A schematic diagram of a training process of a song chord arrangement model according to an embodiment of the present disclosure is schematically shown;

[0129] Fig.11 A flowchart of a song chord arrangement method according to an embodiment of the present disclosure is schematically shown;

[0130] Fig.12 A schematic diagram of a medium according to an embodiment of the present disclosure is schematically shown;

[0131] Fig.13 A block diagram of a training device for a song chord arrangement model according to an embodiment of the present disclosure is schematically shown;

[0132] Fig.14 A block diagram schematically shows a device for arranging song chords according to an embodiment of the present disclosure;

[0133] Fig.15 A schematic diagram of a computing device according to an embodiment of the present disclosure is schematically shown.

[0134] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0135] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0136] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0137] According to an embodiment of the present disclosure, a method, apparatus, medium and computing device for training a song chord arrangement model are proposed.

[0138] In this document, any number of elements in the drawings is used for illustration rather than limitation, and any naming is used only for distinction and does not have any limiting meaning.

[0139] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure. SUMMARY OF THE INVENTION

[0141] The inventors have found that in the current stage of composition work, chord arrangement is usually completed manually by composers, that is, manually writing music scores to complete the chord arrangement work. However, in order to ensure the quality of chord arrangement, manual arrangement usually has high requirements for the arrangers, such as requiring a solid foundation in music theory. Moreover, the process of manual chord arrangement is usually cumbersome and time-consuming, so the arrangement efficiency is generally low and the labor cost is high.

[0142] In order to solve the above problems, the present disclosure provides a training method, device, medium and computing device for a song chord arrangement model. The model training device extracts audio feature information including song chords and a note sequence corresponding to the human voice from an audio file of a sample song, and then generates a note matrix corresponding to the note sequence based on the audio feature information. The matrix is ​​then used as a training sample to perform supervised training on the song chord arrangement model, wherein the extracted song chords are used as sample labels of the training samples. The song chord arrangement model trained in this way can be used by composers for automatic arrangement of song chords.

[0143] By using the above training method, a song chord arrangement model can be provided to composers so that they can use the model to automatically arrange song chords. Through this arrangement method, composers can focus more on the music itself rather than a lot of repetitive work. Not only can it greatly reduce the time required for chord arrangement, thereby improving the arrangement efficiency of composers and reducing labor costs. Moreover, even ordinary users with weak music theory foundation can achieve chord arrangement, which helps to realize more diversified forms of music services.

[0144] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0145] Application Scenario Overview

[0146] It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0147] Figure 1 FIG. 1 is a schematic diagram of the architecture of a song arrangement system provided by an exemplary embodiment. Figure 1 As shown, the system may include a network 10, a server 11, and several electronic devices, such as a mobile phone 12, a mobile phone 13, and a mobile phone 14.

[0148] The server 11 may be a physical server including an independent host, or the server 11 may be a virtual server, a cloud server, etc. hosted by a host cluster. The mobile phones 12-14 are just one type of electronic device that the user may use. In fact, the user may also use electronic devices such as tablet devices, laptop computers, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smart watches, etc.), etc., and one or more embodiments of the present disclosure do not limit this. The network 10 may include various types of wired or wireless networks.

[0149] The server 11 can cooperate with the mobile phones 12-14 to realize chord arrangement. Among them, the server 11 can perform supervised training on the song chord arrangement model based on the sample songs obtained. Furthermore, in one embodiment, the server 11 can deploy the trained song chord arrangement model locally, so that users (such as composers or ordinary users) can use mobile phones 12-14 to upload song arrangement information, and the server 11 will input the received song arrangement information as an input parameter into the song chord arrangement model, and return the song chords output by the model after the arrangement based on the song arrangement information to the corresponding mobile phone for user use. In another embodiment, the server 11 can send the trained song chord arrangement model (such as a lightweight model) to the mobile phones 12-14 and complete the deployment, so that users can use the song chord arrangement model running in the mobile phones 12-14 to arrange the song chords and obtain the song chords output by the model, thereby realizing the lightweight and localized operation of the model. Alternatively, the song chord arrangement model can also be provided to the server 11 after being trained by other devices using the method disclosed in the present invention. In other words, the model training device that is the execution subject of the song chord arrangement model training method disclosed in the present invention can be Figure 1 The server 11 may also be other devices, and the embodiments of the present disclosure are not limited to this.

[0150] According to the training method of the song chord arrangement model described in the embodiment of the present disclosure, the model training device extracts audio feature information including song chords and the note sequence corresponding to the human voice from the audio file of the sample song, and then generates a note matrix corresponding to the note sequence based on the audio feature information, and then uses the matrix as a training sample to perform supervised training on the song chord arrangement model, wherein the extracted song chords are used as sample labels of the training samples. The song chord arrangement model trained in this way can be used by composers to automatically arrange song chords.

[0151] In practical applications, the sample song may be any song containing human voices, wherein the human voices may be the voices of a singer, such as the voices produced when singing the lyrics of a song, etc. Of course, other sounds may also be used, such as the babbling of a baby, the calls of animals, the harmony in a song, etc. Such sounds usually belong to the melody of a song, and are different from the song accompaniment produced by a musical instrument.

[0152] The above-mentioned human voice corresponds to the human voice notes in the song, that is, a continuous human voice lasting a certain length of time in the song corresponds to a human voice note in the song, and the number of human voice notes corresponding to any human voice is related to the pronunciation of this human voice. For example, a Chinese character corresponds to only one human voice note, and an English word corresponds to at least one human voice note, such as "an" corresponds to one human voice note, "apple" corresponds to 2 human voice notes, and "elephant" corresponds to 4 human voice notes, which will not be repeated here.

[0153] Exemplary Methods

[0154] Combine the following Figure 1 For application scenarios, refer to Figure 2 To describe the method for training a song chord arrangement model according to an exemplary embodiment of the present disclosure. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in this regard. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0155] refer to Figure 2 , Figure 2 The flowchart of a method for training a song chord arrangement model according to an embodiment of the present disclosure is schematically shown. The method for training a song chord arrangement model is applied to a model training device and may include the following steps S202-S206.

[0156] Step S202: extracting audio feature information from the audio file of the sample song, wherein the audio feature information includes song chords and a sequence of notes corresponding to human voice.

[0157] In this embodiment, after acquiring the audio file of the sample song, the model training device can extract audio feature information including the song chords and the song mode corresponding to the human voice in the song from the file.

[0158] Among them, the above-mentioned sample songs can be songs maintained by the music consumption platform, and accordingly, their audio files can be obtained from the database for storing song files corresponding to the music consumption platform. Among them, the audio file of any sample song can be a file in the format of WAV (Windows Media Audio), CD (Compact Disc), MIDI (Musical Instrument Digital Interface), etc., and the embodiment of the present disclosure does not limit this. In addition, in order to ensure the arrangement accuracy of the song chord arrangement model obtained by training, sample songs with the same song attributes can be obtained. For example, for the style attribute, the sample songs can all belong to pop music, or all belong to folk songs, or all belong to classical music, etc.; for the lyrics language attribute, the lyrics of the sample songs can all be in Chinese, or all in English, all in Japanese, etc., which will not be repeated. Of course, in order to ensure that the song chord arrangement model has a strong generalization ability, the song attributes of the above-mentioned sample songs can also be different, such as a part of the sample songs belonging to the folk music style, and another part of the sample songs belonging to the classical music style, etc., and the embodiment of the present disclosure does not limit this.

[0159] In one embodiment, the model training device can use MIR (Music Information Retriveal) technology to extract music information from audio files. Specifically, audio processing tools such as Aubio, Essentia or Librosa can be used to extract various audio feature information of sample songs from audio files. The specific use of the above tools can be found in the detailed records in the relevant technology, which will not be repeated here.

[0160] In one embodiment, the model training device may perform the above feature extraction process on a complete audio file of a sample song to extract multiple complete audio feature information of the sample song. Figure 3As shown, the model training device performs feature extraction on the audio file of any sample song. The model training device can extract audio feature information such as song chords, song mode, pitch information and duration information from the audio file. Among them, the song chords are the actual song chords of the sample song; the song mode is the mode of the sample song. In addition, the model training device can directly extract the pitch information and duration information of each vocal note in the audio file. Based on the extracted pitch information and duration information, the model training device can determine each vocal note in the audio file, and each vocal note is sorted according to its position in the sample song timeline, so as to obtain the note sequence corresponding to the human voice in the sample song. Of course, the model training device can also directly extract the above-mentioned note sequence from the audio file, and determine the pitch information and duration information of each vocal note by performing pitch and duration analysis on each vocal note in the sequence. In fact, any vocal note can be represented by the pitch information and duration information of the note.

[0161] In another embodiment, considering that the audio feature information corresponding to different parts of the same sample song may not be the same, such as the first half of a sample song is in the key of C, and the second half is in the key of C#, that is, the song mode is not fixed, then errors may occur when extracting the mode of the entire sample song. Therefore, in order to avoid the above problems and reduce the amount of single calculations in the subsequent matrix generation and model training process, the model training device can also split the complete audio file of a sample song into multiple audio file segments (correspondingly, the note sequence corresponding to the song is also divided into multiple note segments), and extract the corresponding audio feature information segments for each audio file segment. For example, Figure 3 It can also be used to represent the process of extracting features from any audio file segment divided from any sample song. Of course, the features extracted at this time are audio feature information segments corresponding to the audio file segment, such as song chord segments, song mode segments, pitch information segments, and duration information segments.

[0162] Specifically, for any sample song, the model training device can first determine the beats in the sample song, and then divide the audio file of the sample song into multiple audio file segments according to the preset number of beats (correspondingly, the note sequence of the song is also divided into multiple note segments in sequence according to the above-mentioned preset number of beats). The above-mentioned preset number of beats can be the number of beats per measure in the song beat of the sample song - that is, the sample song is divided according to the measures, and each measure corresponds to an audio file segment. For example, a 4 / 4 beat uses a quarter note as one beat, and each measure has four beats. At this time, the model training device can divide the audio file of the sample song in units of 4 beats, thereby obtaining multiple audio file segments corresponding to each measure in the sample song.

[0163] In addition, the note sequence extracted by the above process can be intuitively represented in the form of a piano roll. Figure 3 Any four-beat measure extracted from , the corresponding note fragment is as follows Figure 3 As shown in the piano roll in . The measure includes four beats 301, as shown by vertical lines P1-P4 (P0 is the starting point of the measure and is not included), and correspondingly includes 4 beat intervals (the intervals between any two adjacent vertical lines), corresponding to the time interval of the measure. The note fragment corresponding to the measure extracted through the above process includes 5 vocal notes 302, as shown by N1-N5. The position of any vocal note in the piano roll is determined by the pitch information and duration information of the vocal note: the pitch information determines the ordinate (position height) of the vocal note in the piano roll, such as the pitch of the vocal note N1 is higher than the pitch of N2, and the pitch of N4 is lower than the pitch of N3; the duration information determines the horizontal length of the vocal note in the piano roll, such as the duration of the vocal note N3 is greater than the duration of N2, and the duration of N5 is less than the duration of N4, etc. Of course, the above-mentioned piano roll is only for intuitively illustrating the relationship between various vocal notes. In the actual training process, the model training device may not need to draw the above-mentioned vocal notes in the form of a piano roll.

[0164] The song mode of any song corresponds to the reference pitch of each vocal note in the song, such as the reference pitch of a C-key song is lower than the reference pitch of a C#-key song. The higher the reference pitch of a song, the higher the pitch of each vocal note in the song. It can be seen that the mode of any song matches the pitch information of the vocal notes in the song. In order to make the song chord arrangement model converge quickly during the training process and speed up the training speed, the modes of each sample song used for model training can be controlled to be consistent, such as adjusting the song modes of certain sample songs. Specifically, the model training device can pre-set a standard mode before the start of training, and then after extracting the song mode of the sample song and the pitch information of each vocal note in the note sequence of the song, the model training device can determine whether the song mode of the song is the above-mentioned standard mode, and when the song mode is different from the standard mode, the pitch information of the sample song is adjusted to match the pitch information of the standard mode. The adjusted pitch information can be used to generate a note matrix in the subsequent step 204, which will not be described in detail here.

[0165] For example, when the standard key is preset to C major, if the key of a certain sample song is D major, the key of the song can be adjusted to C major. Figure 4 As shown, each vocal note in the sample song corresponds to a note mark. Figure 4(1) corresponds to the D major before adjustment, and (2) corresponds to the C major after adjustment. Comparing (1) and (2), it can be seen that the pitch of each note mark after adjustment is lowered, and the position of each note mark in the five-line staff is lowered by half a space: for example, the first note mark in (1) is located on the fourth line, and the first note mark in (2) is located between the third and fourth lines. Figure 4 During the adjustment process, the height of each vocal note in the piano roll should be lowered accordingly, which will not be elaborated here.

[0166] In fact, when obtaining sample songs, the song chords can be determined based on the song information, and only sample songs with certain chords can be obtained, such as only songs with major triads or only songs with minor triads, etc. Of course, the song chords of the obtained sample songs can also be adjusted to major triads or minor triads, etc., in the above manner, which will not be repeated here.

[0167] Step S204: generating a note matrix corresponding to the note sequence according to the audio feature information.

[0168] In this embodiment, after extracting the above audio feature information of the sample song, the model training device can generate a note matrix corresponding to the note sequence of the song based on the information. The note matrix can be generated by a multi-dimensional sparse matrix. At this time, the matrix elements corresponding to the vocal notes in the matrix are non-zero elements, and the remaining elements are zero elements. Accordingly, the process of generating the note matrix is ​​the process of determining the position of each non-zero element in the note matrix according to the duration information and pitch information of each vocal note in the note sequence.

[0169] In one embodiment, when the audio feature information includes the duration information of each vocal note in the note sequence, the model training device can determine a beat point sequence including multiple reference beat points in the time interval corresponding to the note sequence with a preset multiple of diaeresis as the basic unit, and then determine the reference note beat points corresponding to each vocal note in the beat point sequence according to the duration information of each vocal note in the sequence. The reference note beat point corresponding to any vocal note is used to determine the matrix element corresponding to the vocal note in the note matrix.

[0170] Among them, the disclosed embodiment does not limit the above-mentioned preset multiples of quavers, for example, it can be eighth notes, sixteenth notes, thirty-second notes, etc. Determine a beat point sequence containing multiple reference beat points in the time interval corresponding to the note sequence according to the preset multiples of quavers, that is, divide a note segment into a preset number of parts (corresponding to the preset multiples). It can be seen that the larger the value of the above-mentioned preset multiples, the more parts a note segment is divided into, so the subsequent adjustment accuracy of the duration information is also higher. Continuing from Figure 3The embodiment of the note segment corresponding to a measure in a 4 / 4 song is shown below. Taking thirty-second notes as an example, Figure 5 Provide explanation.

[0171] like Figure 5 As shown in (1). The beat point sequence is determined with thirty-second notes as the basic unit, that is, the beat interval of 4 beats corresponding to a measure is divided into thirty-two parts. As mentioned above, the measure includes a total of 4 beat intervals, so any beat interval is divided into eight parts, as shown by the dotted lines in the figure. Among them, any dotted line or solid line between P0-P4 represents a reference beat point after division, and all thirty-three reference beat points are arranged in order to form the above beat point sequence.

[0172] For any vocal note, all reference beat points between the two reference beat points located outside its two end points and closest to the corresponding end points are the reference note beat points of the vocal note. Figure 5 As shown in (1), the reference beat point located outside the left end point of the vocal note N2 and closest to the end point is the seventh reference beat point on the right side of P0 (hereinafter referred to as P07), and the reference beat point located outside the right end point of the vocal note N2 and closest to the end point is the fourth reference beat point on the right side of P1 (hereinafter referred to as P14). Therefore, all the reference beat points between P07 and P14 (P07, P1, P11, P12, P13 and P14) are the reference note beat points of the vocal note N2. Other vocal notes are similar to this and will not be repeated. In particular, the left side of the vocal note N1 is already outside P0, so P0 can be used as the first reference note beat point of the vocal note N1.

[0173] After determining the reference note beat points corresponding to each vocal note, the model training device can further determine the matrix elements corresponding to the vocal note in the note matrix according to the reference note beat points determined by the pitch information and duration information corresponding to each vocal note, so as to generate a note matrix. For example, in the case where the above-mentioned note matrix is ​​an m×n matrix, the number of rows m can be the number of notes in the note sequence used to generate the note matrix, and the number of columns n can be the number of beats divided by the beat point sequence according to the preset multiples of the reference beat points. Among them, the above-mentioned preset multiples can be 1 times, 0.5 times, 0.25 times, etc. For example, in the case of dividing the note segment into 32 parts as mentioned above, if the preset multiple is 1 times, then the number of columns of the generated note matrix n=32; if the preset multiple is 0.5 times, then the number of columns of the generated note matrix n=16, which will not be repeated.

[0174] Furthermore, the model training device can determine the rows corresponding to the vocal notes in the note matrix according to the pitch information corresponding to the vocal notes; on the other hand, determine the columns corresponding to the vocal notes in the note matrix according to the reference note beat points corresponding to the vocal notes in the note sequence; and then determine the non-zero elements in the note matrix according to the row and column positions corresponding to the vocal notes, and set the non-zero elements to a preset value to generate a note matrix. The preset value can be 1 to reduce the calculation pressure of the song chord arrangement model.

[0175] From the above process, it can be known that the value of any element in the generated note matrix may be 0 or a non-zero preset value. Moreover, the non-zero rows in the matrix are the rows corresponding to the vocal notes (determined according to the reference note beat points corresponding to the vocal notes), that is, the rows where the preset values ​​are located; the non-zero columns are the columns corresponding to the vocal notes (determined according to the pitch information of the vocal notes), that is, the columns where the preset values ​​are located.

[0176] For the non-zero rows of the note matrix, the model training device can determine the pitch of each vocal note by the MIDI serial number corresponding to the pitch of each vocal note. Figure 6 The 128 MIDI numbers in the MIDI number table shown represent the pitch of each vocal note respectively. For example, if the scale of a vocal note is 4 and the note symbol is D#, then the MIDI number corresponding to the vocal note is 51, which will not be repeated. In this way, the MIDI number corresponding to the pitch of any vocal note may be one of the above 128 numbers. Accordingly, the number of rows of the generated note matrix m=128. Of course, the model training device can also use other numbering rules besides MIDI numbers to represent the pitch information of vocal notes, and the embodiments of the present disclosure are not limited to this.

[0177] For the non-zero columns of the note matrix, the model training device can determine the reference note beat points corresponding to each vocal note. Figure 5 For explanation. Let's assume that the number of columns of the generated note matrix is ​​n=32 columns. For the vocal note N3, the note is located between the reference note beat points P13 and P24, so the vocal note N2 can correspond to the 12th to 20th columns of the matrix. Of course, because its right endpoint is far from P24 and close to P23, the vocal note N2 can also correspond to the 12th to 19th columns of the note matrix. The specific corresponding relationship determination rules can be preset according to actual conditions, and the embodiments of the present disclosure are not limited to this.

[0178] Based on the non-zero rows and columns determined by the above process, the model training device can generate a note matrix based on the sparse matrix through a one-hot algorithm. The specific process can be found in the relevant technology about the algorithm, which will not be described here. Figure 5 In the embodiment shown in (1), because a note sequence is divided into thirty-two parts, the maximum number of columns of the note matrix can be 32. However, it can be understood that the higher the dimension of the note matrix, the greater the amount of matrix calculation in the subsequent model calculation process. Therefore, in order to reduce the amount of calculation of the model calculation and speed up the model prediction speed, the note matrix can be reduced in dimension, that is, the above-mentioned preset multiple can be set to less than 1. Specifically, taking the preset number of bits as 0.5 as an example, the model training device can determine a column in the note matrix through two adjacent reference beat points, thereby generating a 128×16 note matrix. At this time, the 128×16 sparse matrix used can be as follows Figure 7 shown.

[0179] In addition, in order to more accurately determine the reference note beat points corresponding to each vocal note, the model training device can also adjust the note beat points of each vocal note, that is, adjust the duration information of each vocal note according to the above-mentioned reference note beat points. For example, the model training device can adjust the note beat points corresponding to the preset moments of each vocal note in the note sequence to the corresponding nearest reference note beat points; wherein the nearest reference note beat point corresponding to any note beat can be the reference beat point closest to the note beat point in the beat point sequence. The above-mentioned preset moment can be the starting moment of the vocal note (corresponding to the left end point of the vocal note in the piano roll) or the ending moment of the vocal note (corresponding to the right end point of the vocal note in the piano roll).

[0180] For example, for Figure 5 The right end point of the vocal note N1 in (1) is far from P07 and close to P1 (see the partial enlarged view before adjustment as shown in 501). P1 is the nearest reference note beat point on the right side of the vocal note N1. Therefore, the right end point of the vocal note N1 can be extended to P1 (see the partial enlarged view after extension as shown in 502). Specifically, the end time in the duration information of the vocal note N1 can be adjusted to the time corresponding to P1 in the song timeline. Similarly, for Figure 5The left endpoint of the vocal note N2 in (1) is also far from P07 and close to P1 (see 501 for the partial enlarged view before adjustment). P1 is the nearest reference note beat point on the left side of the vocal note N2. Therefore, the left endpoint of the vocal note N1 can be shortened to P1 (see 502 for the partial enlarged view after shortening). Specifically, the start time in the duration information of the vocal note N1 can be adjusted to the time corresponding to P1 in the song timeline. The adjustment process of other vocal notes is similar to this and will not be repeated. After the above adjustment, the endpoints of each vocal note are located at the corresponding outermost reference note beat point. After the above adjustment process, each vocal note strictly corresponds to the reference beat point, which helps to ensure the accuracy of the determined note matrix.

[0181] The adjusted reference note beat points corresponding to each vocal note can be used to generate the matrix elements corresponding to the vocal notes in the note matrix. Specifically, because this adjustment only changes the duration information of the vocal notes, but does not change the pitch information of the vocal notes (i.e. Figure 5 As shown, the horizontal lengths of the endpoints of the vocal notes in the piano roll change, while the vertical height remains unchanged), so for the adjusted vocal notes, the non-zero row corresponding to the note in the note matrix can be determined in the same manner as in the aforementioned embodiment (such as MIDI serial number).

[0182] For non-zero columns, the reference note beat points corresponding to each vocal note can be used to determine the beat points. Taking the generation of a 128×32 note matrix as an example, for Figure 5 The left and right endpoints of the vocal note N2 shown in (2) correspond to the reference note beat points P1 and P14 respectively, so the non-zero columns corresponding to this note in the note matrix are the 9th to 12th columns. Similarly, for Figure 5 The left and right endpoints of the vocal note N3 shown in (2) correspond to the reference note beat points P14 and P23 respectively, so the non-zero columns corresponding to the note in the note matrix can be the 12th to 19th columns. Similar to the above embodiment, a 128×16 note matrix can also be generated. Figure 5 The vocal note N3 shown in (2) corresponds to the non-zero columns from the 13th to the 20th columns in the note matrix. It can be seen that because the right endpoint of the vocal note N3 corresponds to the reference note beat point P23, that is, the note occupies half of the beat interval between the reference beat points P22 to P24, when determining the values ​​of the corresponding column elements of P22 to P24 in the note matrix, the column can be considered as a non-zero column.

[0183] Successor to Figure 5 In the embodiment described above, it is assumed that a 128×16 note matrix is ​​generated, and the non-zero columns in the matrix are as follows: Figure 8The five non-zero value sets shown in the box in the matrix correspond to Figure 5 The five vocal notes N1-N5 shown. Of course, the element values ​​of the other rows in the matrix are all zero. Figure 8 Not shown in the figure.

[0184] As mentioned above, the audio file of the sample song can be divided into multiple audio file segments according to the preset number of beats, and accordingly, the model training device can generate a corresponding note sub-matrix for the audio feature information segment corresponding to each note file segment. Specifically, the model training device can determine the beats in the sample song, and divide the note sequence of the sample song into multiple note segments in sequence according to the preset number of beats, and any note segment corresponds to the audio segment feature information corresponding to the corresponding beat interval in the audio feature information. Furthermore, the model training device can generate note sub-matrices corresponding to each note segment according to the note segment feature information corresponding to each note segment in the note sequence, wherein the chord segment corresponding to any note segment in the song chord extracted in the above steps is used as the sample label of the note sub-matrix corresponding to the note segment. Moreover, the above-mentioned preset number of beats can be the number of beats per measure in the song beat of the sample song, that is, divided according to the measure of the sample song, each measure corresponds to a note segment, and further, the model training device can generate a note sub-matrix corresponding to the note segment.

[0185] Step S206, using the note matrix as a training sample to perform supervised training on the song chord arrangement model, wherein the extracted song chords are used as sample labels of the training samples.

[0186] The note matrix corresponding to the sample song generated in the above steps can be used as a training sample for training the song chord arrangement model. In addition, the song chords extracted in the above step S202 can be used as the sample label of the sample. Specifically, if the complete note matrix corresponding to a sample song is used as a training sample, the sample label of the sample can be the complete song chord corresponding to the song; and when the audio file of the sample song is split into multiple audio file segments, and multiple note sub-matrices are generated based on the audio feature information segment of any audio file segment, any note sub-matrix can be used as a training sample, and the sample label of the sample can be the song chord segment corresponding to the audio file segment.

[0187] As part of the song accompaniment, different chords in the same song often have strong temporal correlation. Therefore, the song chord arrangement model mentioned above can be built based on the reinforcement learning model to take advantage of the inherent feature of the reinforcement model that the current output of the agent will affect the subsequent input of the model, ensuring better arrangement quality for long music with a large number of chords.

[0188] Specifically, Fig. 9 As shown in the figure, the reinforcement learning model mainly consists of four elements: Agent, Environment, Action, and Reward. The operation process of this model is a cyclic process in which the Agent takes Action to change its own State, obtains Reward, and interacts with the Environment. In this process, the Agent receives Observation of the Environment to adjust the next input data according to the Observation.

[0189] The above-mentioned agent may include a first network composed of multiple layers of one-dimensional convolution and a second network composed of multiple fully connected layers. Fig. 9 In a specific structure shown, the first network includes two layers of one-dimensional convolution, which are used to extract features from the note matrix as sample input; the second network consists of ten fully connected layers, which are used to predict and output chord arrangement results based on the features provided by the first network. Of course, the specific network structure of the intelligent agent can be built according to the actual application scenario, and the embodiments of the present disclosure are not limited to this.

[0190] The above embodiments of this specification are explained by taking the example that the number of rows of the note matrix corresponds to the number of notes, and the number of columns of the note matrix corresponds to the reference note beat points corresponding to each note in the measure. However, it can be understood that in the implementation process of the solution, the number of rows of the note matrix can correspond to the reference note beat points corresponding to each note in the measure, and the number of columns of the note matrix can correspond to the number of notes. For example, in a note segment containing 4 sub-notes, the reference beat points are divided based on thirty-second notes, and the reference beat points are divided based on thirty-second notes. Figure 6 When the MIDI serial numbers shown represent the pitch information of each note, a 16×128 note matrix can be generated. Accordingly, the internal structure of the song and chord arrangement model can be adjusted accordingly, and the specific process will not be repeated.

[0191] The song chords of any song usually correspond to the song mode of the song. The chord arrangement result corresponding to any sample can be a multi-dimensional vector, which is used to characterize the probability that the sample output by the song chord arrangement model corresponds to each alternative chord after predicting the training sample input this time. For example, there are 24 chords corresponding to C major, and there may be no chords (or empty chords) at certain moments in the song. Therefore, for a song with the song mode of C major, there are 25 alternative options for any song chord (hereinafter referred to as alternative chords). Fig.10 As shown, the output of the song chord arrangement model can be a 25-dimensional vector, and the 25 element values ​​in the vector correspond to the above 25 alternative chords. Therefore, after inputting any training sample, in the 25-dimensional vector output by the song chord arrangement model, each element value is used to represent the probability value of each alternative chord for the chord predicted by the song chord arrangement model for the input sample. Among them, the alternative chord with the largest probability value can be used as the chord arrangement result output by the model for the sample.

[0192] The goal of reinforcement learning is to build an intelligent agent in a virtual environment and obtain as many cumulative rewards as possible in the calculation iteration. Specifically, in this solution, when the arrangement (action) made by the intelligent agent at a certain chord arrangement point (state State) is consistent with the chord arranged by the composer, it will receive an incentive (reward).

[0193] As mentioned above, the note sequence of a sample song can be divided into multiple note segments, such as dividing the note sequence of a song into multiple note segments according to measures, and then extracting the song chord segments corresponding to each note segment, and generating the note sub-matrices corresponding to each note segment. At this time, each note sub-matrix corresponding to the song can be used as a training sample, and the song chord segment corresponding to each note sub-matrix can be used as the sample label of the sample, and then these (corresponding to the same song) training samples can be input as an episode into the song chord arrangement model to perform supervised training on the model.

[0194] During the model training process, it is usually necessary to calculate the model loss to determine when to terminate the training. Among them, for the above-mentioned model built based on the reinforcement learning model, the model output corresponding reward can be used to represent the model loss. Specifically, after the song chord arrangement model outputs the song chord arrangement result O for any training sample (i.e., any note segment), the reward corresponding to the song chord arrangement result O can be calculated according to the deviation between the song chord arrangement result O and the sample label of the training sample. It can be understood that the larger the reward value corresponding to the chord arrangement result O, the smaller the model loss of the song chord arrangement model at the current moment, and vice versa. Of course, there may not be a strict mathematical relationship between the two.

[0195] In one embodiment, a reward mechanism can be used to calculate the reward for the song chord arrangement result O corresponding to any note segment. For example, when the song chord arrangement result O is equal to the chord segment corresponding to any note segment in the extracted song chords (i.e., O = DL), Reward = a (where a > 0) can be set. However, if only the reward mechanism is used, the model may be trapped in a local optimal solution.

[0196] Therefore, this solution proposes an improved method to calculate the reward for the song chord arrangement result O output by the song chord arrangement model for any note segment. See the following formula (1) or (2):

[0197]

[0198] Among them, a, b, and c in the above formula are all positive numbers, and b < c. The DL is the chord segment corresponding to the any note segment in the extracted song chords, and the key C is the chord set composed of alternative chords corresponding to the song mode. It can be seen that the method described in the above formula (1) or (2) actually introduces a penalty mechanism: not only is the reward given when the song chord arrangement result O output by the song chord arrangement model matches the sample label, but the reward is also recovered when the song chord arrangement result O output by the song chord arrangement model does not match the sample label. Obviously, the relationship between the above song chord arrangement result O and DL and key C can be used to represent the matching degree between the result and the alternative chords in the chord set, and the size of the above reward is positively correlated with this matching degree. By introducing this penalty mechanism, the convergence speed of the model is significantly accelerated, so that the model can understand the chords within the mode and the chords outside the mode more quickly.

[0199] During the above training process, the model training device can calculate whether it is necessary to stop training at any time. For example, as mentioned above, each note sub-matrix corresponding to each note segment corresponding to the same sample song is used as an episode to train the model, and then the model training device can determine whether it is necessary to stop training by calculating the overall reward Reward corresponding to each sample song. For example, the model training device can pre-set the first reward threshold (such as 40) corresponding to the episode and its corresponding first quantity threshold (such as 50 songs). Then the model training device can determine that the arrangement accuracy of the song and chord arrangement model at the current moment meets the requirements when the rewards Reward corresponding to the episodes exceeding the first quantity threshold are not less than the above first reward threshold, thereby stopping the training process. Alternatively, the model training device can also pre-set the second reward threshold (such as 50) corresponding to the episode and its corresponding second quantity threshold (such as 200 songs). Then the model training device can determine that the arrangement accuracy of the song and chord arrangement model at the current moment meets the requirements when the average of the rewards Reward corresponding to multiple episodes exceeding the second quantity threshold is not less than the above second reward threshold, thereby stopping the training process. Among them, the above-mentioned quantity threshold and reward threshold can be set according to the arrangement accuracy index, sample quantity, etc. of the song chord arrangement model. For example, the above-mentioned quantity threshold and reward threshold can be positively correlated with the arrangement accuracy index to ensure that the trained faces have high quality.

[0200] So far, the training process of the song chord arrangement model has been introduced. The song chord arrangement model obtained by the training of the above embodiment can be used by composers to realize the automatic arrangement of song chords. To this end, the embodiment of the present disclosure also proposes a song chord arrangement method, which is applied to an arrangement device, such as Figure 1 The server 11 or the mobile phones 12-14 shown. As shown in 11, the method may include the following steps S1102-S1106.

[0201] Step S1102, obtaining an audio file of a song to be arranged, and extracting a note sequence corresponding to the human voice in the song to be arranged from the audio file.

[0202] First of all, it should be noted that Fig.11 Although some concepts (such as vocal notes, note sequences, note matrices, etc.) in the following embodiments are similar to Figure 2 The corresponding names in the above embodiments are the same, but should be regarded as different concepts: Figure 2 The concept is applied to the training process of the song chord arrangement model, corresponding to the sample song used to train the model; and Fig.11The concept is applied to the application process of the pre-trained song chord arrangement model, corresponding to the song to be arranged by applying the model. Figure 2 and Fig.11 The related embodiments respectively involve concepts with the same name and thus confuse each other.

[0203] In one embodiment, the audio file obtained by the arrangement device can be provided by the user. The user can be a user of the music consumption platform, and the user can record an audio file or receive an audio file sent by other devices through a terminal device running a client of the music consumption platform. Furthermore, the user can arrange chords for the arranged song in a variety of ways. For example, in the case where the arrangement device is the terminal device, the device can be locally deployed with a pre-trained song and chord arrangement model, such as deployed in the above-mentioned client in the form of a functional component, so that the terminal device can arrange chords for the arranged song through the model. For another example, in the case where the arrangement device is the server of the music consumption platform, the user can upload the obtained audio file to the server through the client in the terminal device, so that the server can arrange chords for the arranged song through the locally deployed pre-trained song and chord arrangement model. The song chords obtained by the above arrangement can be used to show the user so that he can view the arrangement effect.

[0204] It is understandable that the above-mentioned song to be arranged is a song that has not yet been arranged with chords, that is, the song contains vocals but does not contain song chords. Specifically, the above-mentioned song file can be a vocal file containing the vocals in the song to be arranged, and the vocal file can be a single-track file extracted from the song to be arranged. Specifically, the vocal file can contain the melody such as the soundtrack and harmony of the song to be arranged, or it can contain no melody and only contain the recorded vocals, and the embodiments of the present disclosure are not limited to this. In this case, the arrangement device can directly extract each vocal note from the vocal file to form a vocal sequence. Alternatively, the above-mentioned song file can also be a MIDI file generated based on the above-mentioned vocal file. In this case, the arrangement device can directly obtain the pitch information and duration information of each vocal note in the song to be arranged based on the MIDI file, and then obtain the note sequence composed of each vocal note.

[0205] In addition, the arrangement device can also extract song feature information such as the song mode of the song to be arranged based on the above audio file. For specific extraction information, please refer to the records of the above embodiment, which will not be repeated here.

[0206] Step S1104, generating a note matrix corresponding to the note sequence according to the note sequence.

[0207] Step S1106, inputting the note matrix into a pre-trained song chord arrangement model, and receiving the song chords output by the song chord arrangement model after arranging the song to be arranged based on the note matrix.

[0208] In this embodiment, after obtaining the note sequence of the song to be arranged, the arrangement device can generate a note matrix according to the note sequence, and use the matrix as the input of the song chord arrangement model, and the model arranges the song to be arranged based on the note matrix to output the song chords. Figure 2 The corresponding model training methods described in the aforementioned embodiments are obtained by pre-training.

[0209] In one embodiment, in order to avoid excessive single-time (a complete calculation process corresponding to one matrix of the input model) calculation amount in the arrangement process of the song and chord arrangement model, the note sequence of the song to be arranged can be split into multiple note segments. For example, the arrangement device can first determine the beats in the song to be arranged, and divide the note sequence into multiple note segments in turn according to the preset number of beats, and then determine the pitch information and duration information of each vocal note of the song to be arranged according to the note sequence, and finally generate the note sub-matrices corresponding to each note segment respectively according to the determined pitch information and duration information. Among them, the above-mentioned preset number of beats can be the number of beats per measure in the song beat of the song to be arranged, that is, it is divided according to the measures of the song to be arranged, and each measure corresponds to a note segment. Further, the arrangement device can generate a note sub-matrix corresponding to the note segment. In this way, the single calculation pressure faced by the song and chord arrangement model can be reduced as much as possible when the song is long. Of course, the above preset number can be consistent with the length of the division of the sample song during the training process of the song chord arrangement model to ensure the accuracy of the chord arrangement of the song to be arranged by the song chord arrangement model.

[0210] Furthermore, in the case where the note sequence is split into multiple note segments, the arrangement device can form a submatrix set with each note submatrix corresponding to each note segment, and input the set into the song chord arrangement model as an episode, and then receive the song chords for this episode output by the model. It can be understood that each note submatrix corresponding to the above episode belongs to the same type, and each note submatrix corresponds to each note segment of the song to be arranged, so the song chords output by the model include the chord segments corresponding to each note segment - the above chord segments together constitute the song chords of the song to be arranged.

[0211] As mentioned above, the song chord arrangement model can be obtained by pre-training with sample songs. Among them, the model training device can extract the note sequence corresponding to the human voice from the audio file of the sample song, and generate the note matrix of the sample song based on the sequence, and then the matrix can be used as a training sample to train the song chord arrangement model. In addition, the model training device can also extract the song chords of the song from the audio file of the sample song, and use the chords as the sample labels of the above training samples (i.e., the note matrix) to achieve supervised training of the model.

[0212] As mentioned above, the song chord arrangement model can be built based on a reinforcement learning model, and the agent in the model can include a first network composed of multiple layers of one-dimensional convolution and a second network composed of multiple fully connected layers. Fig. 9 As shown, after the model receives any input note matrix (or note sub-matrix), the first network in the model can extract feature information of the song to be arranged based on the note matrix, and provide the extracted information to the second network. Correspondingly, the second network can determine the chords corresponding to each vocal note in the note sequence according to the feature information. Among them, each chord together constitutes the song chord of the song to be arranged, and the song chord is the output result of the model, that is, the chord prediction result of the model for the song to be arranged.

[0213] Among them, in the process of training the song chord arrangement model, the model loss loss can be calculated to determine when to stop training. For the above-mentioned model built based on the reinforcement learning model, the reward Reward obtained by the model can be used to represent the model loss loss. For example, when the song chord arrangement model is pre-trained by a sample song, the song chords of the sample song can correspond to multiple alternative chords, and the size of the reward Reward corresponding to the chord arrangement result output by the song chord arrangement model for the sample song can be positively correlated with the degree of matching between the chord arrangement result and the above-mentioned alternative chords. The specific correspondence between the two can be found in the above-mentioned formulas (1) and (2) and the records of related embodiments, which will not be repeated here.

[0214] In one embodiment, the song chord arrangement model can output song chords in a variety of forms. For example, the song chords can be an independent chord file corresponding to the song to be arranged, and the chord file can only contain the chords in the song to be arranged, and the chords are arranged in sequence according to their respective front-to-back order in the timeline of the song to be arranged. For another example, after predicting the chord file, the song chord arrangement model can also merge the chord file into the audio file it received to obtain a complete song audio file corresponding to the song to be arranged. In addition to the chords predicted by the song chord arrangement model, the complete song audio file also includes other music information of the song to be arranged, so it is convenient for comparison and viewing.

[0215] In addition, following the above-mentioned embodiment, when the user provides an audio file, the song chords output by the song chord arrangement model can be used to display to the user. The song chords can be displayed in a visual music score form for the user to view.

[0216] certainly, Fig.11 In the corresponding embodiments above, the specific generation process of the note matrix can be referred to in the above Figure 2 The corresponding embodiments will not be described in detail here.

[0217] Based on the chord arrangement capability of the above-mentioned song chord arrangement model, the music consumption platform can also develop a variety of music service forms, such as allowing ordinary users to arrange music independently, and conducting quality inspection on the arranged chords uploaded by users, etc. The embodiments of the present disclosure do not limit this.

[0218] In summary, according to the training method of the song chord arrangement model of the embodiment of the present disclosure, the model can be used to automatically arrange song chords, so that composers can focus more on the music itself rather than a lot of repetitive work. Not only can the time required for chord arrangement be greatly reduced, thereby improving the arrangement efficiency of composers and reducing labor costs; but even ordinary users with weak music theory foundation can also achieve chord arrangement, which helps to realize more diversified forms of music services.

[0219] Exemplary Media

[0220] After introducing the method of the exemplary embodiment of the present disclosure, next, refer to Fig.12 A medium according to an exemplary embodiment of the present disclosure is described.

[0221] In this exemplary embodiment, the above method can be implemented by a program product, such as a portable compact disk read-only memory (CD-ROM) and including program code, which can be run on a device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, the readable medium 120 can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device.

[0222] The program product may use any combination of one or more readable media. The readable medium 120 may be a readable signal medium or a readable medium. The readable medium 120 may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0223] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0224] The program code contained on the readable medium 120 may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RE, etc., or any suitable combination of the above.

[0225] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0226] Exemplary Devices

[0227] After introducing the medium of the exemplary embodiment of the present disclosure, next, reference is made to Fig.13 A device according to an exemplary embodiment of the present disclosure is described.

[0228] Fig.13 A block diagram of a training device for a song chord arrangement model according to an embodiment of the present disclosure is schematically shown. The device may include:

[0229] The information extraction module 1301 is used to extract audio feature information from the audio file of the sample song, wherein the audio feature information includes song chords and a sequence of notes corresponding to human voice;

[0230] A matrix generation module 1302, used to generate a note matrix corresponding to the note sequence according to the audio feature information;

[0231] The model training module 1303 is used to perform supervised training on the song chord arrangement model using the note matrix as a training sample, wherein the extracted song chords are used as sample labels of the training samples.

[0232] Optionally, the audio feature information further includes duration information of each vocal note in the note sequence, and the device further includes:

[0233] A sequence determination module 1304 is used to determine a beat point sequence including a plurality of reference beat points in a time interval corresponding to the note sequence, taking a preset multiple of dieresis notes as a basic unit;

[0234] The beat point determination module 1305 is used to determine the reference note beat points corresponding to each vocal note in the note sequence according to the duration information of each vocal note. The reference note beat point corresponding to any vocal note is used to determine the matrix element corresponding to the vocal note in the note matrix.

[0235] Optionally, it also includes:

[0236] The beat point adjustment module 1306 is used to adjust the note beat points corresponding to the preset moments of each vocal note in the note sequence to the corresponding nearest reference note beat points; wherein the nearest reference note beat point corresponding to any note beat is the reference beat point closest to the note beat point in the beat point sequence, and the adjusted reference note beat points corresponding to each vocal note are used to generate the matrix elements corresponding to the vocal note in the note matrix.

[0237] Optionally, the preset multiple of the quaver notes is thirty-second notes.

[0238] Optionally, the audio feature information further includes pitch information of each vocal note in the note sequence, and the matrix generation module 1302 is further used for:

[0239] According to the pitch information corresponding to each vocal note and the determined reference note beat point, the matrix element corresponding to the vocal note in the note matrix is ​​determined to generate the note matrix.

[0240] Optionally, the note matrix is ​​an m×n matrix, wherein the number of rows m is the number of notes in the note sequence used to generate the note matrix, and the number of columns n is the number of beats of the beat point sequence divided according to preset multiples of the reference beat point; the matrix generation module 1302 is further used to:

[0241] Determine the row corresponding to each vocal note in the note matrix according to the pitch information corresponding to each vocal note; and

[0242] Determine the columns corresponding to the vocal notes in the note matrix according to the reference note beat points corresponding to the vocal notes in the note sequence;

[0243] The non-zero elements in the note matrix are determined according to the row and column positions corresponding to the various vocal notes, and the non-zero elements are set to preset values ​​to generate the note matrix.

[0244] Optionally, the audio feature information further includes the song mode of the sample song and pitch information of each vocal note in the note sequence, and the pitch information matches the song mode. The device further includes:

[0245] The mode adjustment module 1307 is used to adjust the pitch information to match the pitch information of the standard mode when the mode of the song is different from the standard mode, and the adjusted pitch information is used to generate the note matrix.

[0246] Optionally, the matrix generating module 1302 is further used for:

[0247] Determine the beats in the sample song, and divide the note sequence into a plurality of note segments in sequence according to a preset number of beats, wherein any note segment corresponds to the audio segment feature information corresponding to the corresponding beat interval in the audio feature information;

[0248] Note sub-matrices corresponding to each note fragment are generated according to the note fragment feature information corresponding to each note fragment in the note sequence, and the chord fragment corresponding to any note fragment in the song chord obtained is used as the sample label of the note sub-matrix corresponding to the note fragment.

[0249] Optionally, the number of preset beats is the number of beats per measure in the song beats of the sample song.

[0250] Optionally, the audio feature information further includes the song mode of the sample song;

[0251] The device further includes a reward calculation module, which is used to calculate the reward Reward corresponding to the song chord arrangement result O output by the song chord arrangement model for any note segment through the following formula (1) or (2):

[0252]

[0253] where a, b, and c are all positive numbers, and b < c. DL is the chord segment corresponding to the any note segment in the extracted song chords, and key C is the chord set composed of alternative chords corresponding to the song mode.

[0254] Optionally, the song chord arrangement model is built based on a reinforcement learning model. The agent in the song chord arrangement model includes a first network composed of multiple one-dimensional convolutions and a second network composed of multiple fully connected layers.

[0255] Fig.14 Schematically shows a block diagram of a song chord arrangement device according to an embodiment of the present disclosure. The device may include:

[0256] A file acquisition module 1401, configured to acquire an audio file of a song to be arranged, and extract a note sequence corresponding to the vocals in the song to be arranged from the audio file;

[0257] A matrix generation module 1402, configured to generate a note matrix corresponding to the note sequence according to the note sequence;

[0258] A chord arrangement module 1403, configured to input the note matrix into a pre-trained song chord arrangement model, and receive the song chords output by the song chord arrangement model after arranging the song to be arranged based on the note matrix.

[0259] Optionally, the matrix generation module 1402 is further configured to:

[0260] Determine the beats in the song to be arranged, and sequentially divide the note sequence into multiple note segments according to the number of preset beats;

[0261] Determine the pitch information and duration information of each vocal note in the song to be arranged according to the note sequence;

[0262] The note sub-matrices corresponding to the respective note segments are generated respectively according to the pitch information and duration information of the respective vocal notes.

[0263] Optionally, the chord arrangement module 1403 is further used for:

[0264] Input a sub-matrix set consisting of each note sub-matrix corresponding to each note segment into the song chord arrangement model as an episode;

[0265] The song chords output by the song chord arrangement model are received, wherein the song chords include chord segments corresponding to the respective note segments.

[0266] Optionally, the song chord arrangement model is obtained by pre-training with sample songs, wherein the note matrix of the sample song is used as a training sample, and the song chords of the sample song are used as sample labels of the training sample.

[0267] Optionally, the song and chord arrangement model is built based on a reinforcement learning model, and the intelligent agent Agent in the song and chord arrangement model includes a first network composed of multiple layers of one-dimensional convolution and a second network composed of multiple fully connected layers.

[0268] Optionally, the chord arrangement module 1403 is further used for:

[0269] The first network extracts feature information of the song to be arranged based on the note matrix;

[0270] The second network determines the chords corresponding to the respective vocal notes in the note sequence according to the feature information.

[0271] Optionally, the song chord arrangement model is pre-trained through sample songs, the song mode of the sample songs corresponds to multiple alternative chords, and the size of the reward corresponding to the chord arrangement result output by the song chord arrangement model for the sample song is positively correlated with the degree of matching between the chord arrangement result and the alternative chords.

[0272] Optionally, the audio file includes:

[0273] A vocal file containing the vocals in the song to be arranged; or,

[0274] A MIDI file generated based on a vocal file containing the vocals in the song to be arranged.

[0275] Optionally, the song chords include:

[0276] The independent chord file corresponding to the song to be arranged;

[0277] The independent chord file corresponding to the song to be arranged is merged into the audio file to obtain a complete song audio file.

[0278] Optionally, the audio file is provided by a user, and the song chords are used to be displayed to the user.

[0279] Exemplary Computing Devices

[0280] After introducing the method, medium and apparatus of the exemplary embodiments of the present disclosure, next, reference is made to Fig.15 A computing device according to an exemplary embodiment of the present disclosure is described.

[0281] Fig.15 The computing device 1500 shown is only an example and should not bring any limitation to the functionality and scope of use of the embodiments of the present disclosure.

[0282] like Fig.15 As shown, the computing device 1500 is in the form of a general computing device. The components of the computing device 1500 may include but are not limited to: at least one processing unit 1501, at least one storage unit 1502, and a bus 1503 connecting different system components (including the processing unit 1501 and the storage unit 1502).

[0283] The bus 1503 includes a data bus, a control bus, and an address bus.

[0284] The storage unit 1502 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 15021 and / or a cache memory 15022 , and may further include a readable medium in the form of a non-volatile memory, such as a read-only memory (ROM) 15023 .

[0285] The storage unit 1502 may also include a program / utility 15025 having a set (at least one) of program modules 15024, such program modules 15024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0286] The computing device 1500 may also communicate with one or more external devices 1504 (eg, a keyboard, pointing device, etc.).

[0287] Such communication may be performed via input / output (I / O) interface 1505. In addition, computing device 1500 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via network adapter 1506. Fig.15As shown, network adapter 1506 communicates with other modules of computing device 1500 via bus 1503. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with computing device 1500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0288] It should be noted that, although several units / modules or sub-units / modules of the training device for the song chord arrangement model are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided to be embodied by multiple units / modules.

[0289] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0290] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. A method for training a song chord arrangement model, comprising: Extracting audio feature information from an audio file of a sample song, the audio feature information including song chords, a note sequence corresponding to a human voice, and pitch information of each human voice note in the note sequence; Generating a note matrix corresponding to the note sequence according to the audio feature information, comprising: determining a matrix element corresponding to the vocal note in the note matrix according to the pitch information of each vocal note and the determined reference note beat point, so as to generate the note matrix; The song chord arrangement model is supervisedly trained using the note matrix as a training sample, wherein the extracted song chords are used as sample labels of the training samples.

2. The method according to claim 1, wherein the audio feature information further comprises duration information of each vocal note in the note sequence, and the method further comprises: Taking a preset multiple of diaeresis as a basic unit, determining a beat point sequence including a plurality of reference beat points in a time interval corresponding to the note sequence; The reference note beat points corresponding to the respective vocal notes in the note sequence in the beat point sequence are determined according to the duration information of the respective vocal notes, and the reference note beat points corresponding to any vocal note are used to determine the matrix element corresponding to the vocal note in the note matrix.

3. The method according to claim 2, further comprising: The note beat points corresponding to the preset moments of each vocal note in the note sequence are adjusted to the corresponding nearest reference note beat points; wherein the nearest reference note beat point corresponding to any note beat is the reference beat point closest to the note beat point in the beat point sequence, and the adjusted reference note beat points corresponding to each vocal note are used to generate the matrix elements corresponding to the vocal note in the note matrix.

4. The method according to claim 2 or 3, wherein the preset multiple of the quaver is a thirty-second note.

5. The method according to claim 1, wherein the note matrix is ​​an m×n matrix, wherein: The number of rows m is the number of notes in the note sequence used to generate the note matrix, and the number of columns n is the number of beats of the beat point sequence divided according to the preset multiples of the reference beat point; the matrix element corresponding to the vocal note in the note matrix is ​​determined according to the pitch information corresponding to each vocal note and the determined reference note beat point to generate the note matrix, including: Determine the row corresponding to each vocal note in the note matrix according to the pitch information corresponding to each vocal note; and Determine the columns corresponding to the vocal notes in the note matrix according to the reference note beat points corresponding to the vocal notes in the note sequence; The non-zero elements in the note matrix are determined according to the row and column positions corresponding to the various vocal notes, and the non-zero elements are set to preset values ​​to generate the note matrix.

6. The method according to claim 1, wherein the audio feature information further comprises the song mode of the sample song and the pitch information of each vocal note in the note sequence, and the pitch information matches the song mode, and the method further comprises: When the song mode is different from the standard mode, adjust the pitch information to the pitch information matching the standard mode, and the adjusted pitch information is used to generate the note matrix.

7. The method according to claim 1, wherein generating a note matrix corresponding to the note sequence according to the audio feature information includes: Determine the beats in the sample song, and sequentially divide the note sequence into multiple note segments according to a preset number of beats, and any note segment corresponds to the audio segment feature information corresponding to the corresponding beat interval in the audio feature information; Generate a note sub-matrix corresponding to each note segment according to the note segment feature information corresponding to each note segment in the note sequence, and the chord segment corresponding to any note segment in the extracted song chord is used as the sample label of the note sub-matrix corresponding to the note segment.

8. The method according to claim 7, wherein the preset number of beats is: the number of beats per measure in the song beats of the sample song.

9. The method according to claim 7, wherein the audio feature information further includes the song mode of the sample song; the reward Reward corresponding to the song chord arrangement result O output by the song chord arrangement model for any note segment is calculated by the following formula (1) or (2): in, a, b, and c are all positive numbers, and b < c. DL is the chord segment corresponding to the any note segment in the extracted song chord, and key C is the chord set composed of alternative chords corresponding to the song mode.

10. The method according to claim 1, wherein the song chord arrangement model is built based on a reinforcement learning model, and the agent in the song chord arrangement model includes a first network composed of multiple one-dimensional convolutions and a second network composed of multiple fully connected layers.

11. A method for arranging song chords, comprising: Obtain an audio file of a song to be arranged, and extract the note sequence corresponding to the vocals in the song to be arranged from the audio file; Generate a note matrix corresponding to the note sequence, including: determining the beats in the song to be arranged, and sequentially dividing the note sequence into multiple note segments according to a preset number of beats; determining the pitch information and duration information of each vocal note in the song to be arranged according to the note sequence; generating a note sub-matrix corresponding to each note segment according to the pitch information and duration information of each vocal note; Input the note matrix into a pre-trained song chord arrangement model, and receive the song chords output by the song chord arrangement model after arranging the song to be arranged based on the note matrix.

12. The method according to claim 11, wherein inputting the note matrix into a pre-trained song chord arrangement model, and receiving the song chords output by the song chord arrangement model after arranging the song to be arranged based on the note matrix includes: Input a sub-matrix set consisting of each note sub-matrix corresponding to each note segment into the song chord arrangement model as an episode; The song chords output by the song chord arrangement model are received, wherein the song chords include chord segments corresponding to the respective note segments.

13. The method according to claim 11, wherein the song chord arrangement model is obtained by pre-training with sample songs, wherein: The note matrix of the sample song is used as a training sample, and the song chords of the sample song are used as sample labels of the training sample.

14. According to the method of claim 11, the song chord arrangement model is built based on a reinforcement learning model, and the intelligent agent in the song chord arrangement model includes a first network composed of multiple layers of one-dimensional convolution and a second network composed of multiple fully connected layers.

15. The method according to claim 14, wherein the song chord arrangement model arranges the song to be arranged based on the note matrix, comprising: The first network extracts feature information of the song to be arranged based on the note matrix; The second network determines the chords corresponding to the respective vocal notes in the note sequence according to the feature information.

16. According to the method of claim 14, the song chord arrangement model is obtained by pre-training with sample songs, the song mode of the sample songs corresponds to multiple alternative chords, and the size of the reward corresponding to the chord arrangement result output by the song chord arrangement model for the sample songs is positively correlated with the degree of matching between the chord arrangement result and the alternative chords.

17. The method according to claim 11, wherein the audio file comprises: A vocal file containing the vocals in the song to be arranged; or, A MIDI file generated based on a vocal file containing the vocals in the song to be arranged.

18. The method of claim 11, wherein the song chords include: The independent chord file corresponding to the song to be arranged; The independent chord file corresponding to the song to be arranged is merged into the audio file to obtain a complete song audio file.

19. The method according to any one of claims 11-18, wherein the audio file is provided by a user, and the song chords are used for presentation to the user.

20. A training device for a song chord arrangement model, comprising: An information extraction module is used to extract audio feature information from the audio file of the sample song, wherein the audio feature information includes song chords, a note sequence corresponding to the human voice, and pitch information of each human voice note in the note sequence; A matrix generation module, used to generate a note matrix corresponding to the note sequence according to the audio feature information, specifically used to determine the matrix elements corresponding to the vocal notes in the note matrix according to the pitch information of each vocal note and the determined reference note beat point, so as to generate the note matrix; The model training module is used to perform supervised training on the song chord arrangement model by using the note matrix as a training sample, wherein the extracted song chords are used as sample labels of the training samples.

21. The device according to claim 20, wherein the audio feature information further comprises duration information of each vocal note in the note sequence, and the device further comprises: A sequence determination module, for determining a beat point sequence including a plurality of reference beat points in a time interval corresponding to the note sequence, taking a preset multiple of diaeresis notes as a basic unit; A beat point determination module is used to determine the reference note beat points corresponding to each vocal note in the note sequence in the beat point sequence according to the duration information of each vocal note, and the reference note beat point corresponding to any vocal note is used to determine the matrix element corresponding to the vocal note in the note matrix.

22. The apparatus according to claim 21, further comprising: A beat point adjustment module is used to adjust the note beat points corresponding to the preset moments of each vocal note in the note sequence to the corresponding nearest reference note beat points; wherein the nearest reference note beat point corresponding to any note beat is the reference beat point closest to the note beat point in the beat point sequence, and the adjusted reference note beat points corresponding to each vocal note are used to generate the matrix elements corresponding to the vocal note in the note matrix.

23. The device according to claim 21 or 22, wherein the preset multiple of the quaver notes is thirty-second notes.

24. The device according to claim 23, wherein the note matrix is ​​an m×n matrix, wherein: The number of rows m is the number of notes in the note sequence used to generate the note matrix, and the number of columns n is the number of beats in the beat point sequence divided according to the preset multiples of the reference beat point; the matrix generation module is also used for: Determine the row corresponding to each vocal note in the note matrix according to the pitch information corresponding to each vocal note; and Determine the columns corresponding to the vocal notes in the note matrix according to the reference note beat points corresponding to the vocal notes in the note sequence; The non-zero elements in the note matrix are determined according to the row and column positions corresponding to the various vocal notes, and the non-zero elements are set to preset values ​​to generate the note matrix.

25. The device according to claim 20, wherein the audio feature information further comprises the song mode of the sample song and the pitch information of each vocal note in the note sequence, and the pitch information matches the song mode, and the device further comprises: The mode adjustment module is used to adjust the pitch information to match the pitch information of the standard mode when the mode of the song is different from the standard mode, and the adjusted pitch information is used to generate the note matrix.

26. The apparatus according to claim 20, wherein the matrix generation module is further configured to: Determine the beats in the sample song, and divide the note sequence into a plurality of note segments in sequence according to a preset number of beats, wherein any note segment corresponds to the audio segment feature information corresponding to the corresponding beat interval in the audio feature information; Note sub-matrices corresponding to each note fragment are generated according to the note fragment feature information corresponding to each note fragment in the note sequence, and the chord fragment corresponding to any note fragment in the song chord obtained is used as the sample label of the note sub-matrix corresponding to the note fragment.

27. The device according to claim 26, wherein the number of preset beats is: the number of beats per measure in the song beats of the sample song.

28. The device according to claim 26, wherein the audio feature information further includes the song mode of the sample song; The device further includes a reward calculation module for calculating the reward Reward corresponding to the song chord arrangement result O output by the song chord arrangement model for any note segment through the following formula (1) or (2): in, a, b, and c are all positive numbers, and b < c. DL is the chord segment corresponding to the any note segment in the extracted song chords, and key C is the chord set composed of alternative chords corresponding to the song mode.

29. The device according to claim 20, wherein the song chord arrangement model is built based on a reinforcement learning model, and the agent in the song chord arrangement model includes a first network composed of multiple layers of one-dimensional convolution and a second network composed of multiple fully connected layers.

30. A device for arranging song chords, comprising: A file acquisition module for acquiring an audio file of a song to be arranged and extracting the note sequence corresponding to the vocals in the song to be arranged from the audio file; A matrix generation module for generating a note matrix corresponding to the note sequence according to the note sequence, specifically for determining the beats in the song to be arranged and sequentially dividing the note sequence into multiple note segments according to the number of preset beats; determining the pitch information and duration information of each vocal note in the song to be arranged according to the note sequence; and generating respective note sub-matrices corresponding to the respective note segments according to the pitch information and duration information of each vocal note; A chord arrangement module for inputting the note matrix into a pre-trained song chord arrangement model and receiving the song chords output by the song chord arrangement model after arranging the song to be arranged based on the note matrix.

31. The device according to claim 30, wherein the chord arrangement module is further configured to: Use the set of sub-matrices composed of the respective note sub-matrices corresponding to the respective note segments as an episode to input into the song chord arrangement model; Receive the song chords output by the song chord arrangement model, where the song chords include the chord segments corresponding to the respective note segments.

32. The device according to claim 30, wherein the song chord arrangement model is obtained by pre-training with sample songs, wherein: The note matrix of the sample song is used as a training sample, and the song chords of the sample song are used as the sample labels of the training sample.

33. The device according to claim 30, wherein the song chord arrangement model is built based on a reinforcement learning model, and the agent in the song chord arrangement model includes a first network composed of multiple layers of one-dimensional convolution and a second network composed of multiple fully connected layers.

34. The device according to claim 33, wherein the chord arrangement module is further configured to: The first network extracts the feature information of the song to be arranged based on the note matrix; The second network determines the chords corresponding to each vocal note in the note sequence according to the feature information.

35. According to the device of claim 33, the song chord arrangement model is obtained by pre-training with sample songs, the song mode of the sample songs corresponds to multiple alternative chords, and the size of the reward corresponding to the chord arrangement result output by the song chord arrangement model for the sample song is positively correlated with the degree of matching between the chord arrangement result and the alternative chords.

36. The apparatus of claim 30, wherein the audio file comprises: A vocal file containing the vocals in the song to be arranged; or, A MIDI file generated based on a vocal file containing the vocals in the song to be arranged.

37. The apparatus of claim 30, wherein the song chords include: The independent chord file corresponding to the song to be arranged; The independent chord file corresponding to the song to be arranged is merged into the audio file to obtain a complete song audio file.

38. The apparatus according to any one of claims 30-37, wherein the audio file is provided by a user, and the song chords are used for presentation to the user.

39. A medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 19.

40. A computing device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 19 by running the executable instructions.

Images