Chord recognition model training method and device, chord recognition method and device

Abstract

The present disclosure relates to a training method and device for a chord recognition model, and a chord recognition method and device. The chord recognition method includes: obtaining voiceless audio and drumless audio corresponding to audio; performing feature extraction on the audio, the voiceless audio, and the drumless audio, respectively, to obtain audio features, voiceless audio features, and drumless audio features; by inputting the audio features into a first sub-model in a chord recognition model, inputting the voiceless audio features into a second sub-model in a chord recognition model, and inputting the drumless audio features into a third sub-model in a chord recognition model, to obtain a first chord type, a second chord type, and a third chord type of the audio, respectively; based on the first chord type, the second chord type, and the third chord type of the audio, determine the chord type of the audio. According to the chord recognition method and device of the present disclosure, the accuracy of chord recognition can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of audio and video technology. More specifically, the present disclosure relates to a training method and device for a chord recognition model, and a chord recognition method and device. Background Art

[0002] Automatic chord recognition (ACR) is an important research topic in the field of music information retrieval. Chords are the basis of music formation. They can directly affect the aesthetics of a piece of music, determine the emotion and color of the music, and enrich the content of the music. Common chord types include triads: major triad (maj), minor triad (min), diminished triad (dim), augmented triad (aug), seventh chords: major seventh chord (maj7), dominant seventh chord (7), minor seventh chord (m7), minor major seventh chord (mM7), diminished seventh chord (dim7), augmented seventh chord (aug7), half-diminished seventh chord (half-dim7), as well as additional chords (add), extended chords (extend), and suspended chords (suspend).

[0003] Chord progressions are the basis of pop music, classical music, rock music, blues, jazz and other music genres. In these music genres, melody and rhythm are based on chord progressions. In the songwriting process, songs are usually composed of chord progressions, and then other elements such as melody, bass, and orchestration are considered. Therefore, chord recognition is undoubtedly very important for music learning and composition. For example, in jazz improvisation, jazz bassists can improvise bass according to the chord score, and other solo instrument players also improvise their own solo parts according to the chord score. In the related art, some websites provide some manually annotated music scores for download. However, the chords in these scores need to be manually identified and annotated, and the accuracy of chord recognition is low. Summary of the invention

[0004] The exemplary embodiments of the present disclosure provide a method and device for training a chord recognition model, and a method and device for chord recognition, so as to at least solve the problem of low accuracy of chord recognition in the related art, and may not solve any of the above problems.

[0005] According to an exemplary embodiment of the present disclosure, a training method for a chord recognition model is provided, comprising: obtaining unvoiced audio and undrum-sounded audio corresponding to audio; based on an original audio training set, obtaining an unvoiced audio training set and an undrum-sounded audio training set corresponding to the original audio training set; performing feature extraction on the audio in the original audio training set, the unvoiced audio training set, and the undrum-sounded audio training set, respectively, to obtain an original audio feature set, an unvoiced audio feature set, and an undrum-sounded audio feature set; by inputting the original audio feature set into a first sub-model of a chord recognition model, inputting the unvoiced audio feature set into a second sub-model of the chord recognition model, and inputting the undrum-sounded audio feature set into a The third sub-model is introduced into the chord recognition model, and the predicted chord type of the original audio, the predicted chord type of the audio without voice, and the predicted chord type of the audio without drum sound are obtained respectively; based on the real chord type of the original audio, the predicted chord type of the original audio, the predicted chord type of the audio without voice, and the predicted chord type of the audio without drum sound, the prediction loss of the first sub-model, the prediction loss of the second sub-model, and the prediction loss of the third sub-model are calculated; the parameters of the first sub-model, the parameters of the second sub-model, and the parameters of the third sub-model are adjusted based on the prediction loss of the first sub-model, the prediction loss of the second sub-model, and the prediction loss of the third sub-model respectively.

[0006] Optionally, the step of obtaining the predicted chord type of the original audio by inputting the original audio feature set into the first sub-model in the chord recognition model may include: inputting the original audio feature set into the first sub-model in the chord recognition model, predicting the predicted pitch level, predicted root note and predicted bass of the original audio; and determining the predicted chord type of the original audio based on the predicted pitch level, predicted root note and predicted bass of the original audio.

[0007] Optionally, the step of obtaining the predicted chord type of the unvoiced audio by inputting the unvoiced audio feature set into the second sub-model in the chord recognition model may include: inputting the unvoiced audio feature set into the second sub-model in the chord recognition model, predicting the predicted pitch level, predicted root note and predicted bass of the unvoiced audio; and determining the predicted chord type of the unvoiced audio based on the predicted pitch level, predicted root note and predicted bass of the unvoiced audio.

[0008] Optionally, the step of obtaining the predicted chord type of the audio without drums by inputting the audio feature set without drums into the third sub-model in the chord recognition model may include: inputting the audio feature set without drums into the third sub-model in the chord recognition model to predict a predicted pitch level, a predicted root note and a predicted bass of the audio without drums; and determining the predicted chord type of the audio without drums based on the predicted pitch level, the predicted root note and the predicted bass of the audio without drums.

[0009] Optionally, the step of calculating the prediction loss of the first sub-model, the prediction loss of the second sub-model, and the prediction loss of the third sub-model based on the true chord type of the original audio, the predicted chord type of the original audio, the predicted chord type of the audio without vocals, and the predicted chord type of the audio without drums may include: calculating the prediction loss of the first sub-model based on the true chord type of the original audio and the predicted chord type of the original audio; calculating the prediction loss of the second sub-model based on the true chord type of the original audio and the predicted chord type of the audio without vocals; calculating the prediction loss of the third sub-model based on the true chord type of the original audio and the predicted chord type of the audio without drums.

[0010] Optionally, the step of calculating the prediction loss of the first sub-model based on the real chord type of the original audio and the predicted chord type of the original audio may include: calculating a first chord type difference between the real chord type of the original audio and the predicted chord type of the original audio; calculating a first pitch class difference between the real pitch class of the original audio and the predicted pitch class of the original audio; calculating a first root note difference between the real root note of the original audio and the predicted root note of the original audio; calculating a first bass difference between the real bass of the original audio and the predicted bass of the original audio; and determining the prediction loss of the first sub-model based on the first chord type difference, the first pitch class difference, the first root note difference, and the first bass difference.

[0011] Optionally, the step of calculating the prediction loss of the second sub-model based on the true chord type of the original audio and the predicted chord type of the unvoiced audio may include: calculating a second chord type difference between the true chord type of the original audio and the predicted chord type of the unvoiced audio; calculating a second pitch class difference between the true pitch class of the original audio and the predicted pitch class of the unvoiced audio; calculating a second root note difference between the true root note of the original audio and the predicted root note of the unvoiced audio; calculating a second bass difference between the true bass of the original audio and the predicted bass of the unvoiced audio; and determining the prediction loss of the second sub-model based on the second chord type difference, the second pitch class difference, the second root note difference and the second bass difference.

[0012] Optionally, the step of calculating the prediction loss of the third sub-model based on the real chord type of the original audio and the predicted chord type of the audio without drums may include: calculating a third chord type difference between the real chord type of the original audio and the predicted chord type of the audio without drums; calculating a third pitch class difference between the real pitch class of the original audio and the predicted pitch class of the audio without drums; calculating a third root note difference between the real root note of the original audio and the predicted root note of the audio without drums; calculating a third bass difference between the real bass of the original audio and the predicted bass of the audio without drums; and determining the prediction loss of the third sub-model based on the third chord type difference, the third pitch class difference, the third root note difference, and the third bass difference.

[0013] Optionally, based on the original audio training set, the step of obtaining an audio training set without human voice and an audio training set without drum sound corresponding to the original audio training set may include: performing vocal separation processing on each original audio in the original audio training set to obtain an audio training set without human voice corresponding to the original audio training set; performing drum sound separation processing on each original audio in the original audio training set to obtain an audio training set without drum sound corresponding to the original audio training set.

[0014] According to an exemplary embodiment of the present disclosure, a chord recognition method is provided, comprising: obtaining voiceless audio and drumless audio corresponding to audio; performing feature extraction on the audio, the voiceless audio, and the drumless audio, respectively, to obtain audio features, voiceless audio features, and drumless audio features; obtaining a first chord type, a second chord type, and a third chord type of the audio by inputting the audio features into a first sub-model in a chord recognition model, inputting the voiceless audio features into a second sub-model in the chord recognition model, and inputting the drumless audio features into a third sub-model in the chord recognition model; and determining the chord type of the audio based on the first chord type, the second chord type, and the third chord type of the audio.

[0015] Optionally, the step of obtaining the first chord type of the audio by inputting the audio feature into the first sub-model in the chord recognition model may include: inputting the audio feature into the first sub-model in the chord recognition model to predict the pitch, root note and bass of the audio; and determining the first chord type of the audio based on the pitch, root note and bass of the audio.

[0016] Optionally, the step of obtaining the second chord type of the audio by inputting the unvoiced audio feature into the second sub-model in the chord recognition model may include: inputting the unvoiced audio feature into the second sub-model in the chord recognition model, predicting the pitch, root note and bass of the unvoiced audio feature; and determining the second chord type of the audio based on the pitch, root note and bass of the unvoiced audio feature.

[0017] Optionally, the step of obtaining the third chord type of the audio by inputting the no-drum audio feature into the third sub-model in the chord recognition model may include: inputting the no-drum audio feature into the third sub-model in the chord recognition model, predicting the pitch, root note and bass of the no-drum audio feature; and determining the third chord type of the audio based on the pitch, root note and bass of the no-drum audio feature.

[0018] Optionally, based on the first chord type, the second chord type, and the third chord type of the audio, the step of determining the chord type of the audio may include: calculating the mean of the first chord type, the second chord type, and the third chord type; and determining the chord type of the audio based on the mean.

[0019] Optionally, the step of obtaining audio without human voice and audio without drum sound corresponding to the audio may include: obtaining audio without human voice corresponding to the audio by performing vocal separation processing on the audio; and obtaining audio without drum sound corresponding to the audio by performing drum sound separation processing on the audio.

[0020] Optionally, the chord recognition model may be trained using the chord recognition model training method described in the present disclosure.

[0021] According to an exemplary embodiment of the present disclosure, a training device for a chord recognition model is provided, comprising: a training set acquisition unit, configured to acquire, based on an original audio training set, an audio training set without vocals and an audio training set without drum sounds corresponding to the original audio training set; a feature extraction unit, configured to extract features from the audio in the original audio training set, the audio training set without vocals and the audio training set without drum sounds, respectively, to obtain an original audio feature set, an audio feature set without vocals and an audio feature set without drum sounds; a type prediction unit, configured to input the original audio feature set into a first sub-model of a chord recognition model, input the audio feature set without vocals into a second sub-model of the chord recognition model, and input the audio feature set without drum sounds into the chord recognition model; The third sub-model in the chord recognition model obtains the predicted chord type of the original audio, the predicted chord type of the audio without vocals, and the predicted chord type of the audio without drum sounds, respectively; the loss calculation unit is configured to calculate the predicted loss of the first sub-model, the predicted loss of the second sub-model, and the predicted loss of the third sub-model based on the real chord type of the original audio, the predicted chord type of the original audio, the predicted chord type of the audio without vocals, and the predicted chord type of the audio without drum sounds; and the parameter adjustment unit is configured to adjust the parameters of the first sub-model, the parameters of the second sub-model, and the parameters of the third sub-model based on the predicted loss of the first sub-model, the predicted loss of the second sub-model, and the predicted loss of the third sub-model, respectively.

[0022] Optionally, the type prediction unit may include a first prediction unit configured to: input the original audio feature set into the first sub-model in the chord recognition model, predict the predicted pitch level, predicted root note and predicted bass of the original audio; determine the predicted chord type of the original audio based on the predicted pitch level, predicted root note and predicted bass of the original audio.

[0023] Optionally, the type prediction unit may include a second prediction unit configured to: input the unvoiced audio feature set into the second sub-model in the chord recognition model, predict the predicted pitch level, predicted root note and predicted bass of the unvoiced audio; determine the predicted chord type of the unvoiced audio based on the predicted pitch level, predicted root note and predicted bass of the unvoiced audio.

[0024] Optionally, the type prediction unit may include a third prediction unit configured to: input the drum-free audio feature set into the third sub-model in the chord recognition model to predict the predicted pitch level, predicted root note and predicted bass of the drum-free audio; and determine the predicted chord type of the drum-free audio based on the predicted pitch level, predicted root note and predicted bass of the drum-free audio.

[0025] Optionally, the loss calculation unit may include: a first calculation unit, configured to calculate the prediction loss of the first sub-model based on the real chord type of the original audio and the predicted chord type of the original audio; a second calculation unit, configured to calculate the prediction loss of the second sub-model based on the real chord type of the original audio and the predicted chord type of the audio without vocals; and a third calculation unit, configured to calculate the prediction loss of the third sub-model based on the real chord type of the original audio and the predicted chord type of the audio without drums.

[0026] Optionally, the first calculation unit may be configured to: calculate a first chord type difference between a real chord type of the original audio and a predicted chord type of the original audio; calculate a first pitch class difference between a real pitch class of the original audio and a predicted pitch class of the original audio; calculate a first root note difference between a real root note of the original audio and a predicted root note of the original audio; calculate a first bass difference between a real bass of the original audio and a predicted bass of the original audio; and determine a prediction loss of the first sub-model based on the first chord type difference, the first pitch class difference, the first root note difference, and the first bass difference.

[0027] Optionally, the second computing unit may be configured to: calculate a second chord type difference between the real chord type of the original audio and the predicted chord type of the unvoiced audio; calculate a second pitch class difference between the real pitch class of the original audio and the predicted pitch class of the unvoiced audio; calculate a second root note difference between the real root note of the original audio and the predicted root note of the unvoiced audio; calculate a second bass difference between the real bass of the original audio and the predicted bass of the unvoiced audio; and determine the prediction loss of the second sub-model based on the second chord type difference, the second pitch class difference, the second root note difference and the second bass difference.

[0028] Optionally, the third calculation unit may be configured to: calculate a third chord type difference between the real chord type of the original audio and the predicted chord type of the audio without drums; calculate a third pitch class difference between the real pitch class of the original audio and the predicted pitch class of the audio without drums; calculate a third root note difference between the real root note of the original audio and the predicted root note of the audio without drums; calculate a third bass difference between the real bass of the original audio and the predicted bass of the audio without drums; and determine the prediction loss of the third sub-model based on the third chord type difference, the third pitch class difference, the third root note difference, and the third bass difference.

[0029] Optionally, the training set acquisition unit can be configured to: obtain an audio training set without human voice corresponding to the original audio training set by performing vocal separation processing on each original audio in the original audio training set; obtain an audio training set without drum sound corresponding to the original audio training set by performing drum sound separation processing on each original audio in the original audio training set.

[0030] According to an exemplary embodiment of the present disclosure, there is provided a chord recognition device, comprising: an audio acquisition unit, configured to acquire voiceless audio and drumless audio corresponding to audio; a feature extraction unit, configured to perform feature extraction on the audio, the voiceless audio, and the drumless audio, respectively, to obtain audio features, voiceless audio features, and drumless audio features; a type prediction unit, configured to obtain a first chord type, a second chord type, and a third chord type of the audio by inputting the audio features into a first sub-model in a chord recognition model, inputting the voiceless audio features into a second sub-model in the chord recognition model, and inputting the drumless audio features into a third sub-model in the chord recognition model; and a type determination unit, configured to determine the chord type of the audio based on the first chord type, the second chord type, and the third chord type of the audio.

[0031] Optionally, the type prediction unit may include a first prediction unit configured to: input the audio feature into the first sub-model in the chord recognition model to predict the pitch, root note and bass of the audio; and determine the first chord type of the audio based on the pitch, root note and bass of the audio.

[0032] Optionally, the type prediction unit may include a second prediction unit configured to: input the unvoiced audio feature into the second sub-model in the chord recognition model, predict the pitch class, root note and bass of the unvoiced audio feature; and determine the second chord type of the audio based on the pitch class, root note and bass of the unvoiced audio feature.

[0033] Optionally, the type prediction unit may include a third prediction unit configured to: input the no-drum audio feature into the third sub-model in the chord recognition model, predict the pitch class, root note and bass of the no-drum audio feature; and determine a third chord type of the audio based on the pitch class, root note and bass of the no-drum audio feature.

[0034] Optionally, the type determination unit may be configured to: calculate an average of the first chord type, the second chord type, and the third chord type; and determine the chord type of the audio based on the average.

[0035] Optionally, the audio acquisition unit may be configured to: obtain audio without human voice corresponding to the audio by performing vocal separation processing on the audio; and obtain audio without drum sound corresponding to the audio by performing drum sound separation processing on the audio.

[0036] Optionally, the chord recognition model may be trained using the chord recognition model training method described in the present disclosure.

[0037] According to an exemplary embodiment of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a training method for a chord recognition model or a chord recognition method according to an exemplary embodiment of the present disclosure.

[0038] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor of an electronic device, the electronic device executes a training method for a chord recognition model or a chord recognition method according to an exemplary embodiment of the present disclosure.

[0039] According to an exemplary embodiment of the present disclosure, a computer program product is provided, including a computer program / instruction. When the computer program / instruction is executed by a processor, a training method for a chord recognition model or a chord recognition method according to an exemplary embodiment of the present disclosure is implemented.

[0040] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0041] The three sub-models in the chord recognition model are trained respectively by using the original audio training set, the audio training set without voice, and the audio training set without drum sound to obtain the trained chord recognition model. The trained chord recognition model is then used to perform multi-model chord detection based on sound source separation, thereby improving the accuracy of chord recognition.

[0042] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0044] Figure 1 An exemplary system architecture 100 is shown in which exemplary embodiments of the present disclosure may be applied.

[0045] Figure 2A flow chart showing a method for training a chord recognition model according to an exemplary embodiment of the present disclosure is shown.

[0046] Figure 3 A schematic diagram showing prediction of chord types for audio signals during the training process of a chord recognition model according to an exemplary embodiment of the present disclosure.

[0047] Figure 4 Examples of pitch classes, root notes, and bass notes according to an exemplary embodiment of the present disclosure are shown.

[0048] Figure 5 A flow chart of a chord recognition method according to an exemplary embodiment of the present disclosure is shown.

[0049] Figure 6 A schematic diagram illustrating chord recognition according to an exemplary embodiment of the present disclosure.

[0050] Figure 7 A block diagram showing a training apparatus for a chord recognition model according to an exemplary embodiment of the present disclosure is shown.

[0051] Figure 8 A block diagram illustrating the type prediction unit 73 according to an exemplary embodiment of the present disclosure.

[0052] Fig. 9 A block diagram of the loss calculation unit 74 according to an exemplary embodiment of the present disclosure is shown.

[0053] Fig.10 A block diagram of a chord recognition apparatus according to an exemplary embodiment of the present disclosure is shown.

[0054] Fig.11 is a block diagram of an electronic device 1100 according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0055] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0056] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0057] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three types of parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three types of parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0058] Research on automatic chord recognition has made great progress, especially in small vocabulary. However, there is still room for improvement in recognizing complex chords. Current deep learning automatic chord recognition research focuses more on making neural networks better extract useful chord information from the spectrum, such as improving the model structure, expanding the chord recognition vocabulary, and solving the long-term consistency problem of chord recognition, thus ignoring the role of chords in songs.

[0059] Considering that chords are the cornerstone of the entire music, and the melody and accompaniment that constitute the song are all generated on the basis of chords, and considering that drums will mask the low frequency of the music to a certain extent, and the variability of human voices will bring more chord confusion, the present disclosure proposes a chord recognition method based on blind source separation, firstly separating human voices and drum sounds from the original audio, and retraining the model based on the separated audio to obtain three different models. Then the recognition results of the three models are combined as the final chord recognition result, so as to improve the accuracy of chord recognition.

[0060] Next, we will refer to Figures 1 to 11 A chord recognition model training method and device, and a chord recognition method and device according to exemplary embodiments of the present disclosure are described in detail.

[0061] Figure 1 An exemplary system architecture 100 is shown in which exemplary embodiments of the present disclosure may be applied.

[0062] like Figure 1As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. Users can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages (for example, audio and video data upload requests, chord recognition requests), etc. Various audio applications or video applications may be installed on the terminal devices 101, 102, 103. The terminal devices 101, 102, 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various electronic devices capable of playing, recording, editing, etc. of audio and video, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, etc. When the terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above, and can be implemented as multiple software or software modules (for example, to provide distributed services), or as a single software or software module. No specific limitation is made here.

[0063] In addition, the terminal devices 101, 102, and 103 may also be installed with components for converting electrical signals into sounds (e.g., speakers) to play sounds, and may also be installed with devices for converting analog audio signals into digital audio signals (e.g., microphones) to collect sounds.

[0064] The server 105 may be a server that provides various services, such as a background server that provides support for multimedia applications installed on the terminal devices 101, 102, and 103. The background server may parse, store, and process the received audio and video data upload request, chord recognition request, and other data, and may also receive the audio and video data download request and chord recognition request sent by the terminal devices 101, 102, and 103, and feed back the audio and video data indicated by the audio and video data download request and chord recognition request to the terminal devices 101, 102, and 103.

[0065] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0066] It should be noted that the training method of the chord recognition model and the chord recognition method provided in the embodiments of the present disclosure are usually executed by a terminal device, but can also be executed by a server, or can also be executed by a terminal device and a server in collaboration. Accordingly, the training device of the chord recognition model and the chord recognition device can be set in the terminal device, in the server, or in both the terminal device and the server.

[0067] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers, and the present disclosure has no limitation on this.

[0068] Figure 2 A flow chart showing a method for training a chord recognition model according to an exemplary embodiment of the present disclosure is shown. Figure 3 A schematic diagram showing prediction of chord types for audio signals during the training process of a chord recognition model according to an exemplary embodiment of the present disclosure. Figure 4 Examples of pitch classes, root notes, and bass notes according to an exemplary embodiment of the present disclosure are shown.

[0069] Reference Figure 2 In step S201, based on the original audio training set, an audio training set without vocals and an audio training set without drum sounds corresponding to the original audio training set are obtained. Here, the original audio training set is composed of a plurality of original audios, and the original audio training set is predetermined. Here, the original audio may be any audio that has not been subjected to audio separation, and each original audio may be a segment of audio containing multiple frames. The audio training set without vocals is composed of audio without vocals obtained based on the original audio in the original audio training set, and the audio training set without drum sounds is composed of audio without drum sounds obtained based on the original audio in the original audio training set.

[0070] For example, Figure 3 As shown, the digital audio signal is passed through an audio separation tool to obtain a signal without singing and a signal without drums.

[0071] In step S202, feature extraction is performed on the audio in the original audio training set, the audio training set without voice, and the audio training set without drum sound, respectively, to obtain the original audio feature set, the audio feature set without voice, and the audio feature set without drum sound. Here, any feature extraction method can be used to perform feature extraction, and the present disclosure does not limit this. Here, feature extraction is performed on each audio in the original audio training set, the audio training set without voice, and the audio training set without drum sound, respectively.

[0072] In an exemplary embodiment of the present disclosure, the features of each frame of audio in the three training sets are extracted respectively to obtain three feature sets. In addition, before feature extraction, the audio in the three training sets may be converted into the frequency domain. For example, Figure 3 As shown in the figure, the audio signal can be transformed into the frequency domain by constant-Q Transform (CQT) or variable-Q Transform (VQT), and the feature extraction of the frequency domain transformation result can be performed. CQT / VQT can transform the time domain signal into the frequency domain signal, which is derived from the short-time Fourier transform, but the center frequency is exponentially distributed.

[0073] In step S203, the original audio feature set is input into the first sub-model of the chord recognition model, the non-vocal audio feature set is input into the second sub-model of the chord recognition model, and the non-drum audio feature set is input into the third sub-model of the chord recognition model to obtain the predicted chord type of the original audio, the predicted chord type of the non-vocal audio, and the predicted chord type of the non-drum audio. Here, the predicted chord type of each original audio is obtained by inputting each original audio feature in the original audio feature set into the first sub-model of the chord recognition model, the predicted chord type of each non-vocal audio is obtained by inputting each non-vocal audio feature in the non-vocal audio feature set into the second sub-model of the chord recognition model, and the predicted chord type of each non-drum audio is obtained by inputting each non-drum audio feature in the non-drum audio feature set into the third sub-model of the chord recognition model.

[0074] Here, the first chord type, the second chord type, and the third chord type may be in the form of vectors. The first chord type may be, for example, a vector composed of probabilities that each frame of the original audio belongs to each chord type, the second chord type may be, for example, a vector composed of probabilities that each frame of the audio without vocals belongs to each chord type, and the third chord type may be, for example, a vector composed of probabilities that each frame of the audio without drum sound belongs to each chord type. In this case, the three vectors may be averaged, and the chord type of the audio may be determined based on the vector obtained by the average calculation.

[0075] Here, the first sub-model, the second sub-model, and the third sub-model before training may be the same three models.

[0076] Here, each of the first sub-model, the second sub-model, and the third sub-model can be a neural network. For example, the neural network can be composed of two convolutional layers and one recursive layer. The first convolutional layer can blur the changes and vibrato between each frame and enhance the stability of the musical tone; the second convolutional layer can learn the local characteristics of the tone of each frame. After an activation function, a bidirectional gated recurrent unit (Bi-directional gated recurrent unit, referred to as Bi-GRU) is used to introduce hidden front and back time information of size 128 in each direction, and then the chord type is obtained through post-processing. For example, each of the first sub-model, the second sub-model, and the third sub-model can include the following: Figure 3 The batch normalization layer, two convolutional layers, and Bi-GRU shown in the figure are used to extract the features of each frame of audio. Figure 3 After batch normalization in the first convolutional layer, the second convolutional layer, and the Bi-GRU layer, three feature vectors representing the pitch, root note, and bass of the chord are obtained. Finally, these three feature vectors are combined into the label of the chord type of the frame audio.

[0077] As an example, the predicted pitch, root note, and bass can be as follows: Figure 4 The form shown. Figure 4 In , darker colors indicate higher probabilities. Figure 4 For Db:maj(9) / 3, the probability of the root note being between C and D is the highest, the probability of the bass note being at F is the highest, and the probability of the pitch being between C and D, between D and E, and between F, G and A is the highest.

[0078] In addition, each of the first sub-model, the second sub-model and the third sub-model may further include a feature extraction layer for feature extraction. In this case, the step of extracting features from the original audio training set, the audio training set without vocals and the audio training set without drum sounds may also be performed by the first sub-model, the second sub-model and the third sub-model, respectively. That is, after step S201, the original audio training set may first be input into the first sub-model in the chord recognition model, the audio training set without vocals may be input into the second sub-model in the chord recognition model, and the audio training set without drum sounds may be input into the third sub-model in the chord recognition model, and then the first sub-model, the second sub-model and the third sub-model may perform feature extraction and chord type prediction.

[0079] In an exemplary embodiment of the present disclosure, the chord vocabulary is not limited. The chord vocabulary may be a small vocabulary including only triads, for example.

[0080] In an exemplary embodiment of the present disclosure, the chord vocabulary may have a total of 170 categories, each pitch corresponds to 14 chord types, a total of 12 pitches, including: min, maj, dim, aug, min6, maj6, min7, minmaj7, maj7, 7, dim7, hdim7, sus2, sus4. All chord types are unified within the 14 chord types, regardless of chord inversions and other complex chords. In addition, there are two special symbols, a no chord symbol represented by N and an unknown chord symbol represented by X.

[0081] According to music theory, chords can be represented by pitch, root, and bass. Therefore, three vectors are generated after the Bi-GRU layer to represent the three different features of the chord. Then, these three vectors are connected (for example, the three vectors are concatenated, and the elements in the resulting vector include the probability that the audio belongs to each chord type) to use the Viterbi algorithm to predict the final chord type based on the connected vectors. Specifically, for each frame of audio, the Viterbi algorithm selects the chord type with the highest probability in the connected vector as the chord type of the frame.

[0082] In step S204, based on the true chord type of the original audio, the predicted chord type of the original audio, the predicted chord type of the audio without vocals, and the predicted chord type of the audio without drums, the prediction loss of the first sub-model, the prediction loss of the second sub-model, and the prediction loss of the third sub-model are calculated.

[0083] Here, when calculating the prediction loss of the first sub-model, the prediction error of the original audio can be calculated based on the real chord type of the original audio and the predicted chord type of the original audio for each original audio in the original audio training set, and the prediction error based on each original audio in the original audio training set can be obtained, and then the prediction loss of the first sub-model can be calculated based on the prediction error of each original audio in the original audio training set. That is, for an original audio in the original audio training set, a prediction loss of the first sub-model is calculated. When calculating the prediction loss of the second sub-model, the prediction error of the unvoiced audio can be calculated based on the real chord type of the original audio and the corresponding predicted chord type of the unvoiced audio for each unvoiced audio in the unvoiced audio training set, and the prediction error based on each unvoiced audio in the unvoiced audio training set can be obtained, and then the prediction loss of the second sub-model can be calculated based on the prediction error of each unvoiced audio in the unvoiced audio training set. That is, for an original audio in the unvoiced audio training set, a prediction loss of the second sub-model is calculated. When calculating the prediction loss of the third sub-model, the prediction error of the drum-free audio may be first calculated for each drum-free audio in the drum-free audio training set based on the real chord type of the original audio and the predicted chord type of the corresponding drum-free audio, to obtain the prediction error of each drum-free audio in the drum-free audio training set, and then the prediction loss of the third sub-model may be calculated based on the prediction error of each drum-free audio in the drum-free audio training set. That is, for one original audio in the drum-free audio training set, one prediction loss of the third sub-model is calculated.

[0084] In the present disclosure, any loss calculation method may be used to calculate the predicted loss of the first sub-model, the predicted loss of the second sub-model, and the predicted loss of the third sub-model, and the present disclosure is not limited to this.

[0085] In step S205, the parameters of the first sub-model, the parameters of the second sub-model, and the parameters of the third sub-model are adjusted based on the prediction loss of the first sub-model, the prediction loss of the second sub-model, and the prediction loss of the third sub-model, respectively.

[0086] Here, the first sub-model after training can be used for chord recognition based on the audio to be recognized, the second sub-model after training can be used for chord recognition based on the non-vocal audio corresponding to the audio to be recognized, and the third sub-model after training can be used for chord recognition based on the non-drum audio corresponding to the audio to be recognized.

[0087] In step S202 to step S205, for each original audio in the original audio training set, the loss of the first sub-model is calculated, and the parameters of the first sub-model are adjusted based on the loss; for each non-voice audio in the non-voice audio training set, the loss of the second sub-model is calculated, and the parameters of the second sub-model are adjusted based on the loss; for each non-drum audio in the non-drum audio training set, the loss of the third sub-model is calculated, and the parameters of the third sub-model are adjusted based on the loss. The training of the first sub-model, the second sub-model, and the third sub-model can be performed in parallel.

[0088] After step S205, steps S201 to S205 are repeated until the training end condition is reached. Here, the training end condition may be, for example, that the training reaches a preset number of times, and the loss of at least one of the first sub-model, the second sub-model, and the third sub-model is less than a threshold.

[0089] In an exemplary embodiment of the present disclosure, when obtaining an audio training set without vocals and an audio training set without drum sounds corresponding to the original audio training set based on the original audio training set in step S201, the audio training set without vocals corresponding to the original audio training set can be obtained by performing vocal separation processing on each original audio in the original audio training set, and the audio training set without drum sounds corresponding to the original audio training set can be obtained by performing drum sound separation processing on each original audio in the original audio training set, thereby obtaining an audio training set without vocals and an audio training set without drum sounds corresponding to the original audio training set, so as to facilitate training of a chord recognition model.

[0090] For example, the human voice in the original audio training set is separated from the original audio by an audio separation tool to obtain an audio training set without human voice corresponding to the original audio training set, and the drum sound in the original audio training set is separated from the original audio by an audio separation tool to obtain an audio training set without drum sound corresponding to the original audio training set. Here, the audio separation tool can be, for example, a track separation (spleeter) tool. For example, using the spleeter tool, the human voice and drum sound in the training set are separated from the original audio respectively to obtain audio without human voice and audio without drum sound, and two new data sets are obtained: a training set without human voice and a training set without drum sound.

[0091] Based on the above implementation method, the original audio training set is processed for vocal separation and drum sound separation respectively to obtain an audio training set without vocals and an audio training set without drum sounds corresponding to the original audio training set, thereby obtaining training data for the first sub-model, the second sub-model and the third sub-model.

[0092] In an exemplary embodiment of the present disclosure, when obtaining the predicted chord type of the original audio by inputting the original audio feature set into the first sub-model in the chord recognition model in step S203, the original audio feature set can be first input into the first sub-model in the chord recognition model to predict the predicted pitch level, predicted root note and predicted bass of the original audio, and then the predicted chord type of the original audio is determined based on the predicted pitch level, predicted root note and predicted bass of the original audio, thereby obtaining the predicted chord type of the original audio.

[0093] In an exemplary embodiment of the present disclosure, when obtaining the predicted chord type of the unvoiced audio by inputting the unvoiced audio feature set into the second sub-model in the chord recognition model in step S203, the unvoiced audio feature set may be first input into the second sub-model in the chord recognition model to predict the predicted pitch level, predicted root note and predicted bass of the unvoiced audio, and then the predicted chord type of the unvoiced audio is determined based on the predicted pitch level, predicted root note and predicted bass of the unvoiced audio, thereby obtaining the predicted chord type of the unvoiced audio. The specific processing process is similar to the process of determining the predicted chord type of the original audio.

[0094] In an exemplary embodiment of the present disclosure, when obtaining the predicted chord type of the audio without drums by inputting the audio feature set without drums into the third sub-model in the chord recognition model in step S203, the audio feature set without drums can be first input into the third sub-model in the chord recognition model to predict the predicted pitch level, predicted root note and predicted bass of the audio without drums, and then the predicted chord type of the audio without drums is determined based on the predicted pitch level, predicted root note and predicted bass of the audio without drums, thereby obtaining the predicted chord type of the audio without drums. The specific processing process is similar to the process of determining the predicted chord type of the original audio.

[0095] In an exemplary embodiment of the present disclosure, when the prediction loss of the first sub-model, the prediction loss of the second sub-model, and the prediction loss of the third sub-model are calculated based on the real chord type of the original audio, the predicted chord type of the original audio, the predicted chord type of the audio without vocals, and the predicted chord type of the audio without drum sounds in step S204, the prediction loss of the first sub-model can be calculated based on the real chord type of the original audio and the predicted chord type of the original audio, the prediction loss of the second sub-model can be calculated based on the real chord type of the original audio and the predicted chord type of the audio without vocals, and the prediction loss of the third sub-model can be calculated based on the real chord type of the original audio and the predicted chord type of the audio without drum sounds, thereby improving calculation efficiency.

[0096] In an exemplary embodiment of the present disclosure, when calculating the prediction loss of the first sub-model based on the real chord type of the original audio and the predicted chord type of the original audio, the first chord type difference between the real chord type of the original audio and the predicted chord type of the original audio can be first calculated, the first pitch class difference between the real pitch class of the original audio and the predicted pitch class of the original audio can be calculated, the first root note difference between the real root note of the original audio and the predicted root note of the original audio can be calculated, and the first bass difference between the real bass of the original audio and the predicted bass of the original audio can be calculated. Then, the prediction loss of the first sub-model is determined based on the first chord type difference, the first pitch class difference, the first root note difference and the first bass difference, thereby obtaining the prediction loss of the first sub-model.

[0097] For example, the sum of the first chord type difference, the first pitch difference, the first root difference, and the first bass difference is determined as the prediction loss of the first sub-model. The first chord type difference, the first root difference, and the first bass difference may be sparse categorical crossentropy, and the first pitch difference may be binary crossentropy.

[0098] For example, the sum of the first chord type difference, the first pitch difference, the first root note difference, and the first bass difference may be taken as the prediction loss of the first sub-model.

[0099] In an exemplary embodiment of the present disclosure, when calculating the prediction loss of the second sub-model based on the real chord type of the original audio and the predicted chord type of the unvoiced audio, the second chord type difference between the real chord type of the original audio and the predicted chord type of the unvoiced audio can be first calculated, the second pitch class difference between the real pitch class of the original audio and the predicted pitch class of the unvoiced audio can be calculated, the second root note difference between the real root note of the original audio and the predicted root note of the unvoiced audio can be calculated, and the second bass difference between the real bass of the original audio and the predicted bass of the unvoiced audio can be calculated. Then, the prediction loss of the second sub-model is determined based on the second chord type difference, the second pitch class difference, the second root note difference and the second bass difference, thereby obtaining the prediction loss of the second sub-model.

[0100] For example, the sum of the second chord type difference, the second pitch difference, the second root difference, and the second bass difference is determined as the prediction loss of the second sub-model. The second chord type difference, the second root difference, and the second bass difference may be sparse cross entropy, and the first pitch difference may be binary cross entropy.

[0101] For example, the sum of the second chord type difference, the second pitch difference, the second root note difference, and the second bass difference may be taken as the prediction loss of the second sub-model.

[0102] In an exemplary embodiment of the present disclosure, when calculating the prediction loss of the third sub-model based on the real chord type of the original audio and the predicted chord type of the audio without drums, the third chord type difference between the real chord type of the original audio and the predicted chord type of the audio without drums can be first calculated, the third pitch difference between the real pitch class of the original audio and the predicted pitch class of the audio without drums can be calculated, the third root note difference between the real root note of the original audio and the predicted root note of the audio without drums can be calculated, and the third bass difference between the real bass of the original audio and the predicted bass of the audio without drums can be calculated. Then, the prediction loss of the third sub-model is determined based on the third chord type difference, the third pitch class difference, the third root note difference and the third bass difference, thereby obtaining the prediction loss of the third sub-model.

[0103] For example, the sum of the third chord type difference, the third pitch difference, the third root difference, and the third bass difference is determined as the prediction loss of the third sub-model. The third chord type difference, the third root difference, and the third bass difference are sparse cross entropies, while the third pitch difference is a binary cross entropy.

[0104] For example, the sum of the third chord type difference, the third pitch difference, the third root difference, and the third bass difference may be taken as the prediction loss of the third sub-model.

[0105] Figure 5 A flow chart of a chord recognition method according to an exemplary embodiment of the present disclosure is shown. Figure 6 A schematic diagram illustrating chord recognition according to an exemplary embodiment of the present disclosure.

[0106] Reference Figure 5 In step S501, the audio without human voice and the audio without drum sound corresponding to the audio are obtained. For example, the audio without human voice and the audio without drum sound corresponding to the audio can be obtained by an audio separation tool.

[0107] In step S502, feature extraction is performed on the audio, the audio without human voice, and the audio without drum sound, respectively, to obtain audio features, audio without human voice features, and audio without drum sound features. Here, any feature extraction method can be used to extract features, and the present disclosure does not limit this. In addition, before feature extraction, the audio, the audio without human voice, and the audio without drum sound can be converted into a frequency domain.

[0108] In step S503, the first chord type, the second chord type, and the third chord type of the audio are obtained by inputting the audio feature into the first sub-model of the chord recognition model, inputting the non-voice audio feature into the second sub-model of the chord recognition model, and inputting the non-drum audio feature into the third sub-model of the chord recognition model. Here, the chord recognition model can be trained by the training method of the chord recognition model in the exemplary embodiment of the present disclosure.

[0109] In addition, each of the first sub-model, the second sub-model and the third sub-model may further include a feature extraction layer for feature extraction. In this case, the steps of extracting features from the audio, the audio without human voices and the audio without drum sounds may also be performed by the first sub-model, the second sub-model and the third sub-model, respectively. That is, after step S501, the audio may first be input into the first sub-model in the chord recognition model, the audio without human voices may be input into the second sub-model in the chord recognition model, and the audio without drum sounds may be input into the third sub-model in the chord recognition model, and then the first sub-model, the second sub-model and the third sub-model may perform feature extraction and chord type recognition.

[0110] In step S504, the chord type of the audio is determined based on the first chord type, the second chord type, and the third chord type of the audio.

[0111] Common chord types may include triads: major triad (maj), minor triad (min), diminished triad (dim), augmented triad (aug), seventh chords: major seventh (maj7), dominant seventh (7), minor seventh (m7), minor major seventh (mM7), diminished seventh (dim7), augmented seventh (aug7), half-diminished seventh (half-dim7), as well as additional chords (add), extended chords (extend), and suspended chords (suspend).

[0112] In an exemplary embodiment of the present disclosure, when obtaining audio without human voice and audio without drum sound corresponding to the audio in step S501, the audio without human voice corresponding to the audio can be obtained by performing vocal separation processing on the audio, and the audio without drum sound corresponding to the audio can be obtained by performing drum sound separation processing on the audio, thereby improving the accuracy of the audio without human voice and audio without drum sound corresponding to the audio.

[0113] For example, firstly, the human voice in the audio is separated from the audio by an audio separation tool to obtain audio without the human voice, and then the drum sound in the audio is separated from the audio by an audio separation tool to obtain audio without the drum sound.

[0114] In an exemplary embodiment of the present disclosure, when the first chord type of the audio is obtained by inputting the audio feature into the first sub-model in the chord recognition model in step S503, the audio feature can be firstly input into the first sub-model in the chord recognition model to predict the pitch, root note and bass of the audio, and then the first chord type of the audio is determined based on the pitch, root note and bass of the audio, thereby obtaining the first chord type. For the specific process, refer to the relevant description of step S203.

[0115] In an exemplary embodiment of the present disclosure, when the second chord type of the audio is obtained by inputting the unvoiced audio feature into the second sub-model in the chord recognition model in step S503, the unvoiced audio feature may be first input into the second sub-model in the chord recognition model to predict the pitch, root note and bass of the unvoiced audio feature, and then the second chord type of the audio is determined based on the pitch, root note and bass of the unvoiced audio feature, thereby improving the accuracy of the second chord type. For the specific process, refer to the relevant description of step S203.

[0116] In an exemplary embodiment of the present disclosure, when the third chord type of the audio is obtained by inputting the non-drum audio feature into the third sub-model in the chord recognition model in step S503, the non-drum audio feature can be first input into the third sub-model in the chord recognition model to predict the pitch, root note and bass of the non-drum audio feature, and then the third chord type of the audio is determined based on the pitch, root note and bass of the non-drum audio feature, thereby improving the accuracy of the third chord type. For the specific process, refer to the relevant description of step S203.

[0117] In an exemplary embodiment of the present disclosure, when determining the chord type of the audio based on the first chord type, the second chord type, and the third chord type of the audio in step S504, the mean of the first chord type, the second chord type, and the third chord type may be first calculated, and then the chord type of the audio may be determined based on the mean, thereby improving the accuracy of the chord type. For the specific process, reference may be made to the relevant description of step S204. Here, the first chord type, the second chord type, and the third chord type may be in the form of vectors. The first chord type may be, for example, a vector composed of the probability that each frame of the original audio belongs to each chord type, the second chord type may be, for example, a vector composed of the probability that each frame of the audio without voice belongs to each chord type, and the third chord type may be, for example, a vector composed of the probability that each frame of the audio without drum sound belongs to each chord type. In this case, the three vectors may be averaged, and the chord type of the audio may be determined based on the vector obtained by the average calculation.

[0118] For example, Viterbi decoding may be used to determine the chord type of the audio, wherein the parameters of the Viterbi decoding may be estimated from training set statistics. Figure 6 The result of the average calculation can be input into the Viterbi decoder (or Viterbi algorithm) for Viterbi decoding. The result of the average calculation can be a vector composed of the probabilities that each frame of the original audio belongs to each chord type. For each frame of the original audio, the Viterbi algorithm selects the chord type with the highest probability as the chord type of the frame. The Viterbi algorithm is a dynamic programming algorithm used to find the Viterbi path-implicit state sequence that is most likely to produce the observed event sequence. In automatic chord recognition, the Viterbi algorithm can enhance the continuity and stability of chord recognition.

[0119] The above has been combined Figures 1 to 6 The training method of the chord recognition model and the chord recognition method according to the exemplary embodiment of the present disclosure are described. Figure 7 and Figure 8 A training device for a chord recognition model and its units, and a chord recognition device and its units according to exemplary embodiments of the present disclosure are described.

[0120] Figure 7 A block diagram showing a training apparatus for a chord recognition model according to an exemplary embodiment of the present disclosure is shown. Figure 8 A block diagram illustrating the type prediction unit 73 according to an exemplary embodiment of the present disclosure. Fig. 9 A block diagram of the loss calculation unit 74 according to an exemplary embodiment of the present disclosure is shown.

[0121] Reference Figure 7 The training device of the chord recognition model includes a training set acquisition unit 71, a feature extraction unit 72, a type prediction unit 73, a loss calculation unit 74 and a parameter adjustment unit 75.

[0122] The training set acquisition unit 71 is configured to acquire, based on the original audio training set, an audio training set without human voice and an audio training set without drum sound corresponding to the original audio training set.

[0123] The feature extraction unit 72 is configured to respectively extract features from the original audio training set, the non-vocal audio training set, and the non-drum audio training set to obtain an original audio feature set, a non-vocal audio feature set, and a non-drum audio feature set.

[0124] The type prediction unit 73 is configured to obtain the predicted chord type of the original audio, the predicted chord type of the audio without vocals, and the predicted chord type of the audio without drums, respectively, by inputting the original audio feature set into the first sub-model of the chord recognition model, inputting the unvoiced audio feature set into the second sub-model of the chord recognition model, and inputting the un-drum-sound audio feature set into the third sub-model of the chord recognition model.

[0125] The loss calculation unit 74 is configured to calculate the predicted loss of the first sub-model, the predicted loss of the second sub-model, and the predicted loss of the third sub-model based on the actual chord type of the original audio, the predicted chord type of the original audio, the predicted chord type of the audio without vocals, and the predicted chord type of the audio without drums.

[0126] In an exemplary embodiment of the present disclosure, the training set acquisition unit 71 may be configured to: obtain an audio training set without vocals corresponding to the original audio training set by performing vocal separation processing on each original audio in the original audio training set; obtain an audio training set without drum sounds corresponding to the original audio training set by performing drum sound separation processing on each original audio in the original audio training set.

[0127] In an exemplary embodiment of the present disclosure, Figure 8 As shown, the type prediction unit 73 may include a first prediction unit 731. The first prediction unit 731 may be configured to: input the original audio feature set into the first sub-model in the chord recognition model, predict the predicted pitch level, predicted root note and predicted bass of the original audio; and determine the predicted chord type of the original audio based on the predicted pitch level, predicted root note and predicted bass of the original audio.

[0128] In an exemplary embodiment of the present disclosure, Figure 8 As shown, the type prediction unit 73 may include a second prediction unit 732. The second prediction unit 732 may be configured to: input the unvoiced audio feature set into the second sub-model in the chord recognition model, predict the predicted pitch level, predicted root note and predicted bass of the unvoiced audio; and determine the predicted chord type of the unvoiced audio based on the predicted pitch level, predicted root note and predicted bass of the unvoiced audio.

[0129] In an exemplary embodiment of the present disclosure, Figure 8 As shown, the type prediction unit 73 may include a third prediction unit 733. The third prediction unit 733 may be configured to: input the feature set of the audio without drum sound into the third sub-model in the chord recognition model, predict the predicted pitch level, predicted root note and predicted bass of the audio without drum sound; and determine the predicted chord type of the audio without drum sound based on the predicted pitch level, predicted root note and predicted bass of the audio without drum sound.

[0130] In an exemplary embodiment of the present disclosure, Fig. 9 As shown, the loss calculation unit 74 may include: a first calculation unit 741, configured to calculate the prediction loss of the first sub-model based on the real chord type of the original audio and the predicted chord type of the original audio; a second calculation unit 742, configured to calculate the prediction loss of the second sub-model based on the real chord type of the original audio and the predicted chord type of the audio without vocals; and a third calculation unit 743, configured to calculate the prediction loss of the third sub-model based on the real chord type of the original audio and the predicted chord type of the audio without drums.

[0131] In an exemplary embodiment of the present disclosure, the first calculation unit 741 may be configured to: calculate a first chord type difference between a real chord type of the original audio and a predicted chord type of the original audio; calculate a first pitch class difference between a real pitch class of the original audio and a predicted pitch class of the original audio; calculate a first root note difference between a real root note of the original audio and a predicted root note of the original audio; calculate a first bass difference between a real bass of the original audio and a predicted bass of the original audio; and determine a prediction loss of the first sub-model based on the first chord type difference, the first pitch class difference, the first root note difference, and the first bass difference.

[0132] In an exemplary embodiment of the present disclosure, the second calculation unit 742 may be configured to: calculate a second chord type difference between a real chord type of the original audio and a predicted chord type of the unvoiced audio; calculate a second pitch class difference between a real pitch class of the original audio and a predicted pitch class of the unvoiced audio; calculate a second root note difference between a real root note of the original audio and a predicted root note of the unvoiced audio; calculate a second bass difference between a real bass of the original audio and a predicted bass of the unvoiced audio; and determine the prediction loss of the second sub-model based on the second chord type difference, the second pitch class difference, the second root note difference, and the second bass difference.

[0133] In an exemplary embodiment of the present disclosure, the third calculation unit 743 may be configured to: calculate a third chord type difference between a real chord type of the original audio and a predicted chord type of the audio without drums; calculate a third pitch class difference between a real pitch class of the original audio and a predicted pitch class of the audio without drums; calculate a third root note difference between a real root note of the original audio and a predicted root note of the audio without drums; calculate a third bass difference between a real bass of the original audio and a predicted bass of the audio without drums; and determine a prediction loss of the third sub-model based on the third chord type difference, the third pitch class difference, the third root note difference, and the third bass difference.

[0134] The parameter adjustment unit 75 is configured to adjust the parameters of the first sub-model, the parameters of the second sub-model, and the parameters of the third sub-model based on the prediction loss of the first sub-model, the prediction loss of the second sub-model, and the prediction loss of the third sub-model, respectively.

[0135] Fig.10 A block diagram of a chord recognition apparatus according to an exemplary embodiment of the present disclosure is shown.

[0136] Reference Fig.10 The chord recognition device includes an audio acquisition unit 1001, a feature extraction unit 1002, a type prediction unit 1003 and a type determination unit 1004.

[0137] The audio acquisition unit 1001 is configured to acquire audio without human voice and audio without drum sound corresponding to the audio.

[0138] The feature extraction unit 1002 is configured to perform feature extraction on the audio, the audio without human voice, and the audio without drum sound, respectively, to obtain audio features, audio features without human voice, and audio features without drum sound.

[0139] The type prediction unit 1003 is configured to obtain a first chord type, a second chord type, and a third chord type of the audio by inputting the audio feature into a first sub-model in a chord recognition model, inputting the audio feature without human voice into a second sub-model in a chord recognition model, and inputting the audio feature without drum sound into a third sub-model in a chord recognition model.

[0140] The type determination unit 1004 is configured to determine the chord type of the audio based on the first chord type, the second chord type, and the third chord type of the audio.

[0141] In an exemplary embodiment of the present disclosure, the audio acquisition unit 1001 may be configured to: obtain human voice-free audio corresponding to the audio by performing vocal separation processing on the audio; and obtain drum sound-free audio corresponding to the audio by performing drum sound separation processing on the audio.

[0142] In an exemplary embodiment of the present disclosure, the chord recognition model may be trained by the chord recognition model training method described in the exemplary embodiment of the present disclosure.

[0143] In an exemplary embodiment of the present disclosure, the type prediction unit 1003 may include a first prediction unit (not shown). The first prediction unit may be configured to: input the audio feature into a first sub-model in a chord recognition model, predict the pitch, root note, and bass of the audio; and determine the first chord type of the audio based on the pitch, root note, and bass of the audio.

[0144] In an exemplary embodiment of the present disclosure, the type prediction unit 1003 may include a second prediction unit (not shown). The second prediction unit may be configured to: input the unvoiced audio feature into a second sub-model in a chord recognition model, predict the pitch, root note, and bass of the unvoiced audio feature; and determine the second chord type of the audio based on the pitch, root note, and bass of the unvoiced audio feature.

[0145] In an exemplary embodiment of the present disclosure, the type prediction unit 1003 may include a third prediction unit (not shown). The third prediction unit is configured to: input the non-drum audio feature into a third sub-model in the chord recognition model, predict the pitch, root note and bass of the non-drum audio feature; and determine the third chord type of the audio based on the pitch, root note and bass of the non-drum audio feature.

[0146] In an exemplary embodiment of the present disclosure, the type determination unit 1004 may be configured to: calculate an average of the first chord type, the second chord type, and the third chord type; and determine the chord type of the audio based on the average.

[0147] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0148] The above has been combined Figures 7 to 10 A training device for a chord recognition model and a chord recognition device according to an exemplary embodiment of the present disclosure are described. Fig.11 An electronic device according to an exemplary embodiment of the present disclosure is described.

[0149] Fig.11 is a block diagram of an electronic device 1100 according to an exemplary embodiment of the present disclosure.

[0150] Reference Fig.11 The electronic device 1100 includes at least one memory 1101 and at least one processor 1102. The at least one memory 1101 stores a set of computer executable instructions. When the computer executable instruction set is executed by the at least one processor 1102, the training method of the chord recognition model and the chord recognition method according to the exemplary embodiments of the present disclosure are executed.

[0151] In an exemplary embodiment of the present disclosure, the electronic device 1100 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 1100 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 1100 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0152] In the electronic device 1100, the processor 1102 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0153] The processor 1102 may execute instructions or codes stored in the memory 1101, wherein the memory 1101 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0154] The memory 1101 may be integrated with the processor 1102, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 1101 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 1101 and the processor 1102 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 1102 can read files stored in the memory.

[0155] In addition, the electronic device 1100 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 1100 may be connected to each other via a bus and / or a network.

[0156] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium including instructions is also provided, such as a memory 1101 including instructions, and the above instructions can be executed by a processor 1102 of an electronic device 1100 to complete the above method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0157] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided. The computer program product includes a computer program / instructions. When the computer program / instructions are executed by a processor, a training method for a chord recognition model and a chord recognition method according to an exemplary embodiment of the present disclosure are implemented.

[0158] The above has been referred to Figures 1 to 10 The training method and device of the chord recognition model and the chord recognition method and device according to the exemplary embodiments of the present disclosure are described. However, it should be understood that: Figures 7 to 10 The chord recognition model training device and its unit, the chord recognition device and its unit shown in can be respectively configured as software, hardware, firmware or any combination of the above items to perform specific functions. Fig.11 The electronic device shown in is not limited to including the components shown above, but some components may be added or deleted as needed, and the above components may also be combined.

[0159] According to the training method and device of the chord recognition model of the exemplary embodiment of the present disclosure, firstly based on the original audio training set, an audio training set without voice and an audio training set without drum sound corresponding to the original audio training set are obtained, and then the audio in the original audio training set, the audio training set without voice and the audio training set without drum sound are respectively extracted to obtain the original audio feature set, the audio feature set without voice and the audio feature set without drum sound, and then the original audio feature set, the audio feature set without voice and the audio feature set without drum sound are respectively input into the first sub-model, the second sub-model and the third sub-model of the chord recognition model to obtain The predicted chord type of the original audio, the predicted chord type of the audio without vocals, and the predicted chord type of the audio without drums are obtained; then, based on the real chord type of the original audio, the predicted chord type of the original audio, the predicted chord type of the audio without vocals, and the predicted chord type of the audio without drums, the prediction loss of the first sub-model, the prediction loss of the second sub-model, and the prediction loss of the third sub-model are calculated; then, based on the prediction loss of the first sub-model, the prediction loss of the second sub-model, and the prediction loss of the third sub-model, the parameters of the first sub-model, the second sub-model, and the third sub-model are adjusted respectively, thereby improving the chord recognition effect of the chord recognition model.

[0160] In addition, according to the chord recognition method and device of the exemplary embodiment of the present disclosure, by first acquiring the non-vocal audio and the non-drum audio corresponding to the audio, and then extracting the features of the audio, the non-vocal audio, and the non-drum audio, respectively, to obtain the audio features, the non-vocal audio features, and the non-drum audio features, and by respectively inputting the audio features, the non-vocal audio features, and the non-drum audio features into the first sub-model, the second sub-model, and the third sub-model of the chord recognition model, to obtain the first chord type, the second chord type, and the third chord type of the audio, and then determining the chord type of the audio based on the first chord type, the second chord type, and the third chord type of the audio. According to the chord recognition method and device of the present disclosure, the accuracy of chord recognition can be improved.

[0161] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0162] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for a chord recognition model, characterized in that: include: Based on the original audio training set, obtaining an audio training set without voice and an audio training set without drum sound corresponding to the original audio training set; Performing feature extraction on the audio in the original audio training set, the audio training set without voice, and the audio training set without drum sound, respectively, to obtain an original audio feature set, an audio feature set without voice, and an audio feature set without drum sound; By inputting the original audio feature set into the first sub-model of the chord recognition model, inputting the non-vocal audio feature set into the second sub-model of the chord recognition model, and inputting the non-drum audio feature set into the third sub-model of the chord recognition model, the predicted chord type of the original audio, the predicted chord type of the non-vocal audio, and the predicted chord type of the non-drum audio are respectively obtained, wherein the predicted chord type of the original audio includes a vector composed of the probability that each frame of the original audio belongs to each chord type, the predicted chord type of the non-vocal audio includes a vector composed of the probability that each frame of the non-vocal audio belongs to each chord type, and the predicted chord type of the non-drum audio includes a vector composed of the probability that each frame of the non-drum audio belongs to each chord type; Calculating a prediction loss of the first sub-model based on a true chord type of the original audio and a predicted chord type of the original audio; Calculating a prediction loss of the second sub-model based on the true chord type of the original audio and the predicted chord type of the unvoiced audio; Calculating the prediction loss of the third sub-model based on the true chord type of the original audio and the predicted chord type of the audio without drum sound; Parameters of the first sub-model, parameters of the second sub-model, and parameters of the third sub-model are adjusted based on the prediction loss of the first sub-model, the prediction loss of the second sub-model, and the prediction loss of the third sub-model, respectively.

2. The chord recognition model training method according to claim 1, characterized in that: The step of obtaining the predicted chord type of the original audio by inputting the original audio feature set into the first sub-model in the chord recognition model comprises: Inputting the original audio feature set into the first sub-model in the chord recognition model to predict the predicted pitch level, predicted root note and predicted bass of the original audio; A predicted chord type of the original audio is determined based on the predicted pitch class, the predicted root note, and the predicted bass note of the original audio.

3. The chord recognition model training method according to claim 1, characterized in that: The step of obtaining the predicted chord type of the unvoiced audio by inputting the unvoiced audio feature set into the second sub-model in the chord recognition model comprises: Inputting the unvoiced audio feature set into the second sub-model in the chord recognition model to predict the predicted pitch level, predicted root note and predicted bass of the unvoiced audio; A predicted chord type of the unvoiced audio is determined based on the predicted pitch class, the predicted root note, and the predicted bass of the unvoiced audio.

4. The chord recognition model training method according to claim 1, characterized in that: The step of obtaining the predicted chord type of the audio without drum sound by inputting the audio feature set without drum sound into the third sub-model in the chord recognition model comprises: Inputting the drum-free audio feature set into the third sub-model in the chord recognition model to predict the predicted pitch level, predicted root note and predicted bass of the drum-free audio; A predicted chord type of the audio without drums is determined based on the predicted pitch class, the predicted root note, and the predicted bass note of the audio without drums.

5. The chord recognition model training method according to claim 1, characterized in that: The step of calculating the prediction loss of the first sub-model based on the true chord type of the original audio and the predicted chord type of the original audio comprises: calculating a first chord type difference between a true chord type of the original audio and a predicted chord type of the original audio; Calculating a first pitch level difference between a real pitch level of the original audio and a predicted pitch level of the original audio; Calculating a first root note difference between a true root note of the original audio and a predicted root note of the original audio; calculating a first bass difference between an actual bass of the original audio and a predicted bass of the original audio; A prediction loss of the first sub-model is determined based on the first chord type difference, the first pitch class difference, the first root note difference, and the first bass difference.

6. The chord recognition model training method according to claim 1, characterized in that: The step of calculating the prediction loss of the second sub-model based on the true chord type of the original audio and the predicted chord type of the unvoiced audio comprises: calculating a second chord type difference between a true chord type of the original audio and a predicted chord type of the unvoiced audio; Calculating a second sound level difference between the actual sound level of the original audio and the predicted sound level of the unvoiced audio; Calculating a second root note difference between a true root note of the original audio and a predicted root note of the unvoiced audio; Calculating a second bass difference between the actual bass of the original audio and the predicted bass of the unvoiced audio; A prediction loss of the second sub-model is determined based on the second chord type difference, the second pitch class difference, the second root note difference, and the second bass difference.

7. The chord recognition model training method according to claim 1, characterized in that: The step of calculating the prediction loss of the third sub-model based on the true chord type of the original audio and the predicted chord type of the audio without drum sound comprises: calculating a third chord type difference between a true chord type of the original audio and a predicted chord type of the audio without drum sound; calculating a third pitch level difference between the actual pitch level of the original audio and the predicted pitch level of the audio without the drum sound; Calculating a third root note difference between a true root note of the original audio and a predicted root note of the audio without drum sound; calculating a third bass difference between the actual bass of the original audio and the predicted bass of the audio without the drum sound; A prediction loss of the third sub-model is determined based on the third chord type difference, the third pitch class difference, the third root note difference, and the third bass difference.

8. The chord recognition model training method according to claim 1, characterized in that: Based on the original audio training set, the step of obtaining an audio training set without voice and an audio training set without drum sound corresponding to the original audio training set comprises: By performing voice separation processing on each original audio in the original audio training set, an audio training set without voice corresponding to the original audio training set is obtained; By performing drum sound separation processing on each original audio in the original audio training set, an audio training set without drum sound corresponding to the original audio training set is obtained.

9. A chord recognition method, characterized in that: include: Get the audio without voice and the audio without drum sound corresponding to the audio; Extract features of the audio, the audio without voice, and the audio without drum sound respectively to obtain audio features, audio features without voice, and audio features without drum sound; By inputting the audio feature into a first sub-model of a chord recognition model, inputting the audio feature without vocals into a second sub-model of a chord recognition model, and inputting the audio feature without drum sounds into a third sub-model of a chord recognition model, a first chord type, a second chord type, and a third chord type of the audio are obtained respectively, wherein the first chord type includes a vector composed of probabilities that each frame of audio of the original audio belongs to each chord type, the second chord type includes a vector composed of probabilities that each frame of audio without vocals belongs to each chord type, and the third chord type includes a vector composed of probabilities that each frame of audio without drum sounds belongs to each chord type; A chord type of the audio is determined based on the first chord type, the second chord type, and the third chord type of the audio.

10. The chord recognition method according to claim 9, characterized in that: The step of obtaining a first chord type of the audio by inputting the audio feature into the first sub-model in the chord recognition model comprises: Input the audio features into the first sub-model in the chord recognition model to predict the pitch, root note and bass of the audio; A first chord type of the audio is determined based on a pitch class, a root note, and a bass note of the audio.

11. The chord recognition method according to claim 9, characterized in that: The step of obtaining a second chord type of the audio by inputting the unvoiced audio feature into the second sub-model in the chord recognition model comprises: Inputting the unvoiced audio feature into the second sub-model in the chord recognition model to predict the pitch, root note and bass of the unvoiced audio feature; A second chord type of the audio is determined based on the pitch class, root note, and bass of the unvoiced audio feature.

12. The chord recognition method according to claim 9, characterized in that: The step of obtaining the third chord type of the audio by inputting the non-drum sound audio feature into the third sub-model in the chord recognition model comprises: Inputting the non-drum sound audio feature into the third sub-model in the chord recognition model to predict the pitch, root note and bass of the non-drum sound audio feature; A third chord type of the audio is determined based on the pitch class, root note, and bass of the no-drum audio feature.

13. The chord recognition method according to claim 9, characterized in that: Based on the first chord type, the second chord type, and the third chord type of the audio, the step of determining the chord type of the audio comprises: Calculate the mean of the first chord type, the second chord type, and the third chord type; A chord type of the audio is determined based on the mean.

14. The chord recognition method according to claim 9, characterized in that: The steps of obtaining the non-voice audio and non-drum audio corresponding to the audio include: By performing voice separation processing on the audio, an audio without voice corresponding to the audio is obtained; By performing drum sound separation processing on the audio, audio without drum sound corresponding to the audio is obtained.

15. The chord recognition method according to any one of claims 9 to 14, characterized in that: The chord recognition model is obtained by training the chord recognition model training method described in any one of claims 1 to 8.

16. A training device for a chord recognition model, characterized in that: include: A training set acquisition unit is configured to acquire, based on the original audio training set, an audio training set without voice and an audio training set without drum sound corresponding to the original audio training set; A feature extraction unit is configured to perform feature extraction on the audio in the original audio training set, the audio training set without voice, and the audio training set without drum sound, respectively, to obtain an original audio feature set, an audio feature set without voice, and an audio feature set without drum sound; a type prediction unit configured to obtain a predicted chord type of the original audio, a predicted chord type of the audio without vocals, and a predicted chord type of the audio without drums, respectively, by inputting the original audio feature set into a first sub-model of a chord recognition model, inputting the audio feature set without vocals into a second sub-model of the chord recognition model, and inputting the audio feature set without drums into a third sub-model of the chord recognition model, wherein the predicted chord type of the original audio includes a vector composed of probabilities that each frame of the original audio belongs to each chord type, the predicted chord type of the audio without vocals includes a vector composed of probabilities that each frame of the audio without vocals belongs to each chord type, and the predicted chord type of the audio without drums includes a vector composed of probabilities that each frame of the audio without drums belongs to each chord type; A first calculation unit, configured to calculate a prediction loss of the first sub-model based on a true chord type of the original audio and a predicted chord type of the original audio; A second calculation unit is configured to calculate the prediction loss of the second sub-model based on the real chord type of the original audio and the predicted chord type of the unvoiced audio; A third calculation unit configured to calculate the prediction loss of the third sub-model based on the true chord type of the original audio and the predicted chord type of the audio without drum sound; and A parameter adjustment unit is configured to adjust the parameters of the first sub-model, the parameters of the second sub-model and the parameters of the third sub-model based on the prediction loss of the first sub-model, the prediction loss of the second sub-model and the prediction loss of the third sub-model, respectively.

17. The chord recognition model training device according to claim 16, characterized in that: The type prediction unit includes a first prediction unit configured to: Inputting the original audio feature set into the first sub-model in the chord recognition model to predict the predicted pitch level, predicted root note and predicted bass of the original audio; A predicted chord type of the original audio is determined based on the predicted pitch class, the predicted root note, and the predicted bass note of the original audio.

18. The chord recognition model training device according to claim 16, characterized in that: The type prediction unit includes a second prediction unit configured to: Inputting the unvoiced audio feature set into the second sub-model in the chord recognition model to predict the predicted pitch level, predicted root note and predicted bass of the unvoiced audio; A predicted chord type of the unvoiced audio is determined based on the predicted pitch class, the predicted root note, and the predicted bass of the unvoiced audio.

19. The chord recognition model training device according to claim 16, characterized in that: The type prediction unit includes a third prediction unit configured to: Inputting the drum-free audio feature set into the third sub-model in the chord recognition model to predict the predicted pitch level, predicted root note and predicted bass of the drum-free audio; A predicted chord type of the audio without drums is determined based on the predicted pitch class, the predicted root note, and the predicted bass note of the audio without drums.

20. The chord recognition model training device according to claim 16, characterized in that: The first computing unit is configured to: calculating a first chord type difference between a true chord type of the original audio and a predicted chord type of the original audio; Calculating a first pitch level difference between a real pitch level of the original audio and a predicted pitch level of the original audio; Calculating a first root note difference between a true root note of the original audio and a predicted root note of the original audio; calculating a first bass difference between an actual bass of the original audio and a predicted bass of the original audio; A prediction loss of the first sub-model is determined based on the first chord type difference, the first pitch class difference, the first root note difference, and the first bass difference.

21. The chord recognition model training device according to claim 16, characterized in that: The second computing unit is configured to: calculating a second chord type difference between a true chord type of the original audio and a predicted chord type of the unvoiced audio; Calculating a second sound level difference between the actual sound level of the original audio and the predicted sound level of the unvoiced audio; Calculating a second root note difference between a true root note of the original audio and a predicted root note of the unvoiced audio; Calculating a second bass difference between the actual bass of the original audio and the predicted bass of the unvoiced audio; A prediction loss of the second sub-model is determined based on the second chord type difference, the second pitch class difference, the second root note difference, and the second bass difference.

22. The chord recognition model training device according to claim 16, characterized in that: The third computing unit is configured to: calculating a third chord type difference between a true chord type of the original audio and a predicted chord type of the audio without drum sound; calculating a third pitch level difference between the actual pitch level of the original audio and the predicted pitch level of the audio without the drum sound; Calculating a third root note difference between a true root note of the original audio and a predicted root note of the audio without drum sound; calculating a third bass difference between the actual bass of the original audio and the predicted bass of the audio without the drum sound; A prediction loss of the third sub-model is determined based on the third chord type difference, the third pitch class difference, the third root note difference, and the third bass difference.

23. The chord recognition model training device according to claim 16, characterized in that: The training set acquisition unit is configured as follows: By performing voice separation processing on each original audio in the original audio training set, an audio training set without voice corresponding to the original audio training set is obtained; By performing drum sound separation processing on each original audio in the original audio training set, an audio training set without drum sound corresponding to the original audio training set is obtained.

24. A chord recognition device, characterized in that: include: An audio acquisition unit configured to acquire audio without human voice and audio without drum sound corresponding to the audio; A feature extraction unit is configured to extract features from the audio, the audio without voice, and the audio without drum sound, respectively, to obtain audio features, audio features without voice, and audio features without drum sound; a type prediction unit, configured to obtain a first chord type, a second chord type, and a third chord type of the audio by inputting the audio feature into a first sub-model of a chord recognition model, inputting the non-vocal audio feature into a second sub-model of the chord recognition model, and inputting the non-drum sound audio feature into a third sub-model of the chord recognition model, respectively, wherein the first chord type includes a vector composed of probabilities that each frame of audio of the original audio belongs to each chord type, the second chord type includes a vector composed of probabilities that each frame of audio of the non-vocal audio belongs to each chord type, and the third chord type includes a vector composed of probabilities that each frame of audio of the non-drum sound audio belongs to each chord type; and The type determination unit is configured to determine the chord type of the audio based on the first chord type, the second chord type, and the third chord type of the audio.

25. The chord recognition device according to claim 24, characterized in that: The type prediction unit includes a first prediction unit configured to: Input the audio features into the first sub-model in the chord recognition model to predict the pitch, root note and bass of the audio; A first chord type of the audio is determined based on a pitch class, a root note, and a bass note of the audio.

26. The chord recognition device according to claim 24, characterized in that: The type prediction unit includes a second prediction unit configured to: Inputting the unvoiced audio feature into the second sub-model in the chord recognition model to predict the pitch, root note and bass of the unvoiced audio feature; A second chord type of the audio is determined based on the pitch class, root note, and bass of the unvoiced audio feature.

27. The chord recognition device according to claim 24, characterized in that: The type prediction unit includes a third prediction unit configured to: Inputting the non-drum sound audio feature into the third sub-model in the chord recognition model to predict the pitch, root note and bass of the non-drum sound audio feature; A third chord type of the audio is determined based on the pitch class, root note, and bass of the no-drum audio feature.

28. The chord recognition device according to claim 24, characterized in that: The type determination unit is configured to: Calculate the mean of the first chord type, the second chord type, and the third chord type; A chord type of the audio is determined based on the mean.

29. The chord recognition device according to claim 24, characterized in that: The audio acquisition unit is configured as follows: By performing voice separation processing on the audio, an audio without voice corresponding to the audio is obtained; By performing drum sound separation processing on the audio, audio without drum sound corresponding to the audio is obtained.

30. The chord recognition device according to any one of claims 24-29, characterized in that: The chord recognition model is obtained by training the chord recognition model training method described in any one of claims 1 to 8.

31. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method as claimed in any one of claims 1 to 15.

32. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor of an electronic device, the electronic device is caused to perform the method according to any one of claims 1 to 15.

33. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 15 is implemented.

Images