An improvisational accompaniment generation device

Abstract

This invention discloses an improvisational accompaniment generation device, comprising: a melody detection unit for detecting melody in acquired audio data to obtain a melody, wherein the melody is divided by beats; a chord arrangement unit for arranging chords based on a CRF-based chord arrangement model according to the melody notes of multiple consecutive beats, to obtain and store the chords corresponding to the melody notes of multiple beats; a chord prediction unit for predicting chords based on a CRF-based chord prediction model according to the melody notes of multiple consecutive beats, the chords corresponding to the melody notes of multiple beats, and the measure number corresponding to the melody notes of the current beat, to predict the chords of the next beat in real time; and an accompaniment generation unit for automatically matching a playing mode to the predicted chords according to the pitch change rate, and then generating an accompaniment based on the predicted chords and the matched playing mode. This device can provide improvisational accompaniment for melodies in real time and ensure the harmony of the improvisational accompaniment.

Description

Technical Field

[0001] This invention belongs to the field of automatic music composition technology, specifically relating to an improvisational accompaniment generation device. Background Technology

[0002] Automatic composition refers to using algorithmic models to find suitable chords for a given melody to accompany it. Existing automatic composition systems can only generate songs after the melody has been recorded, and cannot interact with the singer in real time. Therefore, there is an urgent need for an improvisational accompaniment system that can generate accompaniment in real time.

[0003] Existing real-time accompaniment models can be broadly categorized into data-driven and rule-based models, employing various algorithms such as Hidden Markov Models (HMMs) and reinforcement learning. However, these algorithms all have shortcomings in terms of musicality, interactivity, and immediacy. Balancing the immediacy and musicality of the accompaniment is the biggest challenge. To provide real-time improvisational accompaniment for performers, the computational complexity of the algorithm should not be too high, thus reducing the load on computing devices. However, a simple algorithm can lead to a decrease in the harmony between the generated accompaniment and melody, resulting in a poorer auditory experience.

[0004] To address the issue of poor real-time accompaniment performance, some researchers have employed simple data matching methods for rapid generation. For example, patent document CN104380371A discloses a device for automatically generating real-time accompaniment from input music data. This device extracts music data information through an analyzer, then generates a query against a music database using a query generator. Finally, it matches suitable music from the music database based on the query information to quickly generate the accompaniment. However, this method requires a large music database and a connection to a cloud database, making it highly susceptible to network interference. This results in temporal fluctuations in the accompaniment, unstable real-time performance, and the rigidity of the generated music due to the reliance on music data queries.

[0005] Besides the methods mentioned above, some researchers have optimized the search process using algorithms or simplified machine learning models as much as possible. For example, the paper "A Real-Time Accompaniment Generation System for Popular Music Based on Human Vocal Melody" uses a lightweight ResNet18 to extract melodies from human voices and then uses a simple Markov chain for computation. These methods can achieve real-time chord arrangement tasks, but they all result in a delay of more than one beat.

[0006] To address the issue of excessive delays in real-time chord arrangement, some scholars have proposed using chord prediction methods for chord generation. However, this leads to the problem of exposure bias (Nan Jiang et al. 2020). That is, if the next time step is predicted based on the previously predicted sequence, any error in the previous prediction can have devastating consequences for subsequent generation. Summary of the Invention

[0007] In view of the above, the purpose of this invention is to provide an improvisational accompaniment generation device that can provide improvisational accompaniment for melodies in real time and ensure the harmony of the improvisational accompaniment.

[0008] To achieve the above-mentioned objectives, the improvisational accompaniment generation device provided in the embodiments includes:

[0009] A melody detection unit is used to perform melody detection on the acquired audio data to obtain a melody, wherein the melody is divided by beat;

[0010] The chord arrangement unit is used to arrange chords based on the chord arrangement model built on CRF according to the melody notes of multiple consecutive beats, so as to obtain the arranged chords corresponding to the melody notes of multiple beats and store them as a queue.

[0011] The chord prediction unit is used to predict chords in real time by using a chord prediction model built based on CRF, based on the melody notes of multiple consecutive beats, the chord arrangements corresponding to the melody notes of multiple beats, and the measure number corresponding to the melody notes of the current beat.

[0012] The accompaniment generation unit is used to automatically match the playing pattern for the predicted chords based on the pitch change rate and cadence detection results, and then generate an accompaniment based on the predicted chords and the matched playing pattern.

[0013] In one embodiment, for the chord arrangement model, the melodic notes of each input beat are processed by feature extraction based on the arrangement feature template, and the extracted arrangement features are then input into the chord arrangement model.

[0014] The melodic notes of the rhythm are feature-extracted based on the arrangement feature template, including:

[0015] Based on the current total number of beats N, the note with the longest duration within the Nth, N-1th, N-2th, ..., NM-1th beats is taken as the extracted melody feature to obtain an arrangement feature composed of M melody features. Preferably, the value of M is 8.

[0016] In one embodiment, for a chord prediction model, each input data is processed by feature extraction based on a prediction feature template, and the extracted prediction features are then input into the chord prediction model; wherein, the input data includes melodic notes of multiple beats, the arranged chords corresponding to the melodic notes of multiple beats, and the measure number corresponding to the melodic notes of the current beat;

[0017] Feature extraction is performed on the input data based on the predicted feature template, including:

[0018] Based on the current total number of beats N, the note with the longest duration in the melody of the (N-1), (N-2), ..., (NM)th beats is taken as the extracted melody feature;

[0019] Based on the current total number of beats N, the chords corresponding to the N-1, N-2, ..., NM beats are taken as the extracted chord features;

[0020] The measure number corresponding to the melody note of this beat is used as the measure feature;

[0021] The prediction features consist of M melody features, M chord features, and 1 measure feature. Preferably, the value of M is 8.

[0022] In one embodiment, the chord arrangement model and the chord prediction model need to be subjected to supervised learning using training samples to optimize the model parameters before they are applied.

[0023] The process of constructing training samples includes: cleaning the acquired music dataset, extracting and sampling melody and chord data, shifting the sampled data by an octave, and unifying the tonality.

[0024] The extraction and sampling of melody and chord data includes: converting the cleaned music file into a MIDI file of a melody track; extracting the main melody track data from the MIDI file and converting it into two-dimensional data including a timing array and a pitch array; sampling the pitch according to the timing information to extract the melody scale information and obtain a one-dimensional timing array of the melody.

[0025] The data with chord markings are extracted separately from the cleaned music files, and the chord pitch information at the same time point is converted into component notes in MIDI pitch form. The chords are stored according to the component notes, resulting in two-dimensional data including the component note pitch array and timing data. The chord information of the component notes is sampled according to the timing information to obtain a one-dimensional timing array of the chord component notes.

[0026] In one embodiment of the accompaniment generation unit, automatically matching a playing mode for the predicted chords based on the pitch change rate and cadence detection results includes:

[0027] The absolute value of the pitch difference between the current note and the previous note is calculated based on the input chord to determine the pitch change rate. The pitch change rate is then used to determine whether the current musical phrase belongs to the verse or chorus type, so as to switch between the verse playing mode and the chorus playing mode.

[0028] When it is a verse type, there are multiple preset verse playing modes for chord matching;

[0029] When it is a chorus type, there are multiple preset chorus playing modes for chorus matching;

[0030] By detecting chord cadences and determining whether a musical phrase has ended, when a musical phrase is detected in progress, if it is a verse type, multiple matching verse regular playing patterns are played alternately; if it is a chorus type, multiple matching chorus regular playing patterns are played alternately.

[0031] When the end of a musical phrase is detected, the phrase will be played in either the verse embellishment mode or the chorus embellishment mode as a transition between phrases.

[0032] Preferably, the verse playing mode and the chorus playing mode can be played using piano, guitar, bass, or cello;

[0033] The playing mode is described as [the number of notes in the chord from low to high, start time (beat), duration (beat), instrument name].

[0034] In one embodiment of the improvisational accompaniment generation unit, the detection and determination of whether a musical phrase has ended by chord cadence detection includes:

[0035] Starting with the chord of the current beat, trace back to the chords of the previous K beats. If a cadence / half-cadence chord progression can be formed, then chord phrasing is performed, and the musical phrase is considered to have ended.

[0036] In one embodiment, the text input for the playing mode is described as [the nth note of the chord components from low to high, start time (beat), duration (beat), instrument name].

[0037] This converts the text format of the playing mode into a digital format, which has a one-to-one relationship with the MIDI command. For example, the digital format is 0:0->1|0,0=>1|0, where 0: means wait 0 / 16 beats before executing this line, -> means set the instrument used by the track, so 0->1 means set track 1 to instrument 0, |0 means wait 0 milliseconds before executing this statement, and => means play the specified note, so 0=>1 means play the 0th note on track 1.

[0038] In one embodiment, the improvisational accompaniment generation device further includes a reverberation unit for noise reduction processing of the audio data collected from human voices, and the noise-reduced audio data is input to the melody detection unit for melody detection.

[0039] It is also used to mix and output the accompaniment generated by the accompaniment generation unit with the noise-reduced audio data.

[0040] In one embodiment, the improvisational accompaniment generation device further includes:

[0041] The human voice acquisition unit is used to acquire audio data of human voices;

[0042] The playback unit is used to play the audio data with accompaniment output by the reverb unit;

[0043] The visual editing unit allows users to control the start or stop of the accompaniment and configure parameters through a visual interface. These parameters include: key, metronome speed (beat speed), zero-crossing rate, minimum recognition volume, whether to turn on the metronome and metronome interval, whether to filter all semitones, whether to filter the starting semitone, chord harmony, and chord constraints.

[0044] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0045] The improvisational accompaniment generation device provided in this embodiment divides accompaniment generation into a chord arrangement stage and a chord prediction stage. It innovatively proposes a new combination model for these two stages: a chord arrangement model and a chord prediction model. This allows for real-time chord generation within 0.05 seconds while maintaining acceptable accuracy. Regarding harmonic performance, the accompaniment generation unit offers multiple accompaniment playing modes that automatically switch in real-time based on the characteristics of the input melody and the generated harmony, providing users with a superior accompaniment experience. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a schematic diagram of the improvisational accompaniment generation device provided in the embodiment;

[0048] Figure 2 This is a flowchart of pitch recognition provided in the embodiment;

[0049] Figure 3 This is a diagram showing the contents of an .mxl format music file provided in the embodiment;

[0050] Figure 4 This is a schematic diagram of the chord arrangement model provided in the embodiment;

[0051] Figure 5 This is a schematic diagram of the chord prediction model provided in the embodiment;

[0052] Figure 6 This is a flowchart of feature extraction based on configured feature templates and predicted feature templates provided in the embodiment;

[0053] Figure 7 This is the verification process for the chord prediction model provided in the implementation example;

[0054] Figure 8 This is a visual example of the MIDI rollout window for the accompaniment mode provided in the embodiment;

[0055] Figure 9 This is a schematic diagram of the accompaniment mode switching process provided in the embodiment;

[0056] Figure 10 This is an example diagram of the staff notation for the accompaniment mode provided in the embodiment;

[0057] Figure 11 This is a flowchart of the accompaniment generation process provided in the embodiment;

[0058] Figure 12 This is a flowchart of an embodiment of an improvisational accompaniment generation device. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0060] To address the issues of real-time performance and harmony in existing accompaniment devices, this embodiment provides a method such as... Figure 1 The improvisational accompaniment generation device shown includes a melody detection unit, a chord arrangement unit, a chord prediction unit, and an accompaniment generation unit. These four units work together to generate chords in real time and provide real-time improvisational accompaniment for the melody, while ensuring the harmony of the improvisational accompaniment.

[0061] For the melody detection unit

[0062] The melody detection unit is used to perform melody detection on the acquired audio data to obtain a melody, which is divided by beat. In this embodiment, the melody detection process includes a pitch recognition stage and a tonality recognition stage.

[0063] like Figure 2As shown, the pitch recognition stage includes: First, calculating the zero-crossing rate of the input audio data. Based on the magnitude of the zero-crossing rate, it is determined whether the input audio data corresponds to a human voice. If the zero-crossing rate exceeds a threshold, it is considered noise, and the output pitch and loudness are both 0; otherwise, the pitch recognition process proceeds. Then, pitch recognition is performed, including: To reduce spectral data leakage, the audio data is windowed, and a three-dimensional spectrogram is constructed using short-time Fourier transform (where the horizontal axis represents time, the vertical axis represents frequency, and the depth of each point represents the loudness at the current moment). Due to noise, the constructed spectrogram contains many spikes and is not suitable for direct pitch recognition. Therefore, the spectrogram needs to be converted into an envelope spectrum, that is, connecting the peaks of loudness at a certain moment with a curve. Simultaneously, points with slopes greater than a certain value are treated as noise points and filtered out based on the slope of each peak endpoint, thus greatly improving noise resistance. The frequencies corresponding to the peaks of the filtered envelope spectrum are taken as all frequencies existing at the current moment. Among these frequencies, one is the fundamental tone, and the rest are overtones. Because overtones are integer multiples of the fundamental frequency, this characteristic can be used to calculate and identify the frequency of the fundamental tone, thus obtaining the pitch and loudness (i.e., volume) of the current fundamental tone. Finally, the volume and pitch of the fundamental tone are subjected to Kalman filtering to further reduce the influence of noise, bringing it closer to its true value, and whether human voice has been input is determined based on whether the volume reaches a threshold.

[0064] The pitch identified during the pitch recognition stage is the absolute pitch, which needs to be converted to the corresponding melody based on the tonality. Therefore, in the tonality recognition stage, the pitch is first converted into a scale format using the frequency and scale conversion formula: t = round(log2(f / f0)) * 12 + 57, where round() means rounding to the nearest integer, f is the current pitch, f0 is 440Hz, 57 is the scale corresponding to 440Hz, and t is the number corresponding to the calculated scale.

[0065] In this embodiment, if a fixed tonality is set, the scale is directly converted into the fixed-tonality simplified musical notation. If automatic tonality recognition is set, it is necessary to utilize the rule that most music rarely contains semitones and that at least two notes of the tonic chord of the key appear in the first measure. Specifically, automatic tonality recognition includes: first, performing a modulo 12 operation on each identified scale to obtain pitv, and accumulating it in the pitv position of noteCount; then, converting the 12 notes in noteCount into simplified musical notation in 12 keys, counting the number of semitones and assigning them to the corresponding key positions in baseNoteProb, finding the set of keys with the fewest semitones in baseNoteProb, and selecting the key from the set whose first measure melody has the most notes of the tonic chord of that key as the determined key; finally, converting the obtained melody into the simplified musical notation of the identified scale in that key.

[0066] For chord arrangement unit and chord prediction unit

[0067] The chord arrangement unit uses a CRF-based chord arrangement model to arrange chords based on melodic notes from multiple consecutive beats, obtaining arranged chords corresponding to the melodic notes of multiple beats and storing them in a queue. The chord prediction unit uses a CRF-based chord prediction model to predict chords based on melodic notes from multiple consecutive beats, the arranged chords corresponding to the melodic notes of multiple beats, and the measure number corresponding to the melodic note of the current beat, in order to predict the chord of the current beat in real time. Specifically, the chord arrangement model uses the melodic notes of the previous M beats (e.g., 8 beats) to arrange chords for the melodic notes of the previous beat. The arranged chords are stored in a queue to provide more accurate chord input for the chord prediction model. The chord prediction model uses the first M beats (e.g., 8 beats), the chords corresponding to the first M beats (e.g., 8 beats) provided by the chord arrangement model, and measure features as input. The model uses the Viterbi algorithm to calculate and predict the possible chords and their probabilities for the current beat. Then, it filters according to the user-set requirements of whether to filter all semitones or the starting semitone, harmony, and chord constraints. Finally, it selects the chord with the highest probability from all candidates that meet the rules for accompaniment playback, thereby achieving the effect of real-time accompaniment.

[0068] In this embodiment, the chord arrangement model and chord prediction model need to undergo supervised learning using training samples to optimize model parameters before being applied. The process of constructing training samples includes: sequentially cleaning the acquired music dataset, extracting and sampling melody and chord data, octave shifting of the sampled data, and tonality unification.

[0069] The acquired music dataset contains sheet music for 6672 songs, stored as .mxl format music files. These files can be opened with software such as Musescore. Figure 3 As shown, each exemplary song file contains not only melody notes, title, lyrics, and beat information, but also the key of the song and accompaniment chord markings, making it ideal for training real-time chord arrangement models and chord prediction models.

[0070] Dataset cleaning includes removing music pieces that do not conform to the beat count of the training samples, such as pieces with a three-beat rhythm, thus ensuring that all music in the dataset is in 4 / 4 or 2 / 4 time, facilitating subsequent 16:4 sampling of melodies and chords. It also includes parsing the music files, reading the start time of each chord, and subtracting the start time of the next chord from the start time of each chord to calculate the duration of each chord, thereby finding the shortest chord duration in the music. If the shortest chord duration of a piece is less than one beat, the piece is discarded. This ensures the effectiveness of the training samples, because in beat-based sampling, chords with excessively short durations can cause the model to learn incorrect chord progressions.

[0071] Melody and chord data extraction and sampling refers to extracting melody and chord data from music files and labeling them with one-dimensional time-series arrays to facilitate model training and chord generation. Specifically, this involves: converting the cleaned music file into a MIDI file containing a melody track. The MIDI file consists of several sub-data items with the same format, each recording all information of a track in a multi-track format. The main melody track data is extracted from the MIDI file and converted into two-dimensional data including time-series and pitch arrays. Since two-dimensional data is not suitable for direct model training, pitch is sampled according to the time-series information and according to sixteenth notes (i.e., quarter beats) to extract the melody scale information, resulting in a one-dimensional time-series array of the melody. The pitch is labeled using the MIDI scale notation method; for example, middle C on a piano corresponds to a MIDI value of 60. In this embodiment, the cleaned music file is parsed to obtain score data. The key information is read from the score data for later key unification. Then, data with chord markings are extracted separately from the score data, and the chord pitch information at the same time point is converted into constituent notes in MIDI pitch form. For example, C4, E4, G4 appearing at time 1 and [60, 64, 67], [62, 65, 69] appearing at time 2 are stored according to their constituent notes, resulting in two-dimensional data including a constituent note pitch array and timing data. This method of not directly storing chord names from the MXL file can preserve chord information to the greatest extent, cover a richer range of chords, and facilitate subsequent octave shifting and key unification operations. The chord information of the constituent notes is sampled according to the timing information (quarter note, i.e., one beat) to obtain a one-dimensional timing array of the chord constituent notes.

[0072] Octave shifting is performed to improve training performance and avoid auditory confusion caused by generated harmonies and melodies being in the same octave. The sampled data above was shifted by octaves. The process includes: defining the standard octave of chords as 48 points in the MIDI protocol (i.e., the 4th row of octaves) and the standard octave of melodies as 72 points in the MIDI protocol (i.e., the 6th row of octaves); counting the octave segments where each note in the melody and chord is located, and obtaining the octave containing the most data; calculating the distance between the octave data and the corresponding standard octave of the melody or chord; and shifting the melody and chord data to obtain music data unified to the standard octave.

[0073] Tonality unification refers to adjusting the interval between a piece of music and C major if it is in a major key, and between it and A minor if it is in a minor key. This converts all major music to C major and all minor music to A minor, thereby reducing training costs and improving training effectiveness.

[0074] In this embodiment, the data that has undergone data cleaning, extraction and sampling of melody and chord data, octave shifting of the sampled data, and tonality unification is saved to a txt text file in the format of filename {tonality|[melody]|[chord component notes]}, which is convenient for reading during model training.

[0075] In this embodiment, both the chord arrangement model and the chord prediction model are built upon Conditional Random Fields (CRFs). A CRF is an undirected graphical model. Assuming X and Y represent the joint distribution random variables of the observation sequence to be labeled and the corresponding labeled sequence, respectively, then the Conditional Random Field (X, Y) is an undirected graphical model conditioned on the observation sequence. In this embodiment, the CRF model is modified to conform to the form of musical composition, and then the two modified chord arrangement models and the chord prediction model are combined to generate improvisational accompaniment. Specifically, the chord arrangement model, based on the melody notes collected from the previous eight beats, uses probability calculations to derive the chord arrangement corresponding to the previous beat and stores it in a queue container for the chord prediction model to use. The chord prediction model uses the melody notes from the previous eight beats and the chord arrangement generated by the chord arrangement model from the previous eight beats to predict the chord for the current beat and plays it. Schematic diagrams of these two models are shown below. Figure 4 and Figure 5 As shown. Figure 4 and Figure 5 In this context, n represents the input melody note. k to n k-7 This indicates that the chord arrangement model is trained using the melodic notes of the current beat and the previous seven beats. c represents the arrangement chord corresponding to each beat. Cur represents the input data of the chord prediction model, including the melodic note n of each beat, the arrangement chord c of each beat, and the measure feature T. Pre represents the predicted chord for the current beat, and null represents an empty string.

[0076] In this embodiment, based on the requirements of the chord arrangement task, the applicant designed the following format as the input data format, and examples of the format are shown in Table 1. Specifically, when obtaining the longest-lasting note within a beat, if every note within a beat is different, then the note with the longest duration within that beat is extracted. Additionally, blank lines are used to segment each piece of music.

[0077] Table 1

[0078] The note with the longest duration within a beat (x1) This beat chord (as model output, y) Location 4 4-6-1 Upper Seven Beats 6 4-6-1 Upper Six Beats 2 2-4-6 Five beats 4 2-4-6 The first four beats 3 6-1-3 Upper three beats 3 6-1-3 The first two shots 1 6-1-3 Previous shot 5 1-3-5 Current shot

[0079] In this embodiment, since the chord prediction task is more complex than chord arrangement, the input data is also more complex. A sample data format for the chord prediction model is shown in Table 2. Unlike chord arrangement model 1, the chord prediction model adds chords as input. This input, in actual operation, uses the output of chord arrangement instead of the model's own chord output. This is because, in actual operation, the predicted chords may not harmonize with the melody of the current beat due to their predictive nature; therefore, the chord arrangement model needs to correct previously predicted chords. Additionally, the newly added measure feature is a sequential arrangement of measure numbers from 1 to 8, used to indicate whether the chords change every four beats within the same measure, and can also be used as constraints for the chord template.

[0080] Table 2

[0081]

[0082] Because the appearance of melodies and chords in a piece of music is somewhat random and related, relying solely on melodic structure analysis is unlikely to yield good generation results. Therefore, it is necessary to simultaneously mine the preceding and following relationships between melodies and chords. CRF models can express long-distance preceding and following dependencies through feature templates and effectively integrate various related or unrelated information. It is important to note that the preceding and following relationship is defined by an observation window consisting of the current output chord and the preceding melody notes or chords. Theoretically, a longer window allows for more observed feature information, but this can lead to reduced training and runtime efficiency, and even overfitting. Conversely, a window that is too small results in insufficient feature information acquisition, affecting the chord generation quality. Therefore, after weighting, this CRF model will select features from the preceding 8 beats for analysis at most, and a univariate melodic feature template has been designed.

[0083] For the chord arrangement task, since only the relationship between melody and chord is considered, only a univariate feature template for the melody is designed to extract arrangement features as input to the chord arrangement model. It's also important to note that the input for training the arrangement model differs from the input for prediction by one beat. This is because arrangement requires the corresponding chord of the melody at that specific time, while the output for model training can only be the chord corresponding to the current beat. Therefore, training requires using the longest-lasting note within the current beat. During prediction, shifting the input forward one beat generates the arrangement chord for the previous beat. The design of the arrangement feature template is shown in Table 3.

[0084] Table 3

[0085]

[0086]

[0087] In this embodiment, features are extracted from the melody notes of the current beat and the previous seven beats using an arrangement feature template. The extracted arrangement features are then input into the chord arrangement model. The extraction process begins with the current beat of the melody note and proceeds forward 0, 1, 2, 3, ..., 7 beats, using the longest-lasting note within the melody as the extracted melody feature. This results in an arrangement feature composed of eight melody features. Specifically, for this arrangement feature template, the longest-lasting note is transformed into a binary function for feature extraction. When the feature function takes a specific value, the arrangement feature template is instantiated. For example, using the training data shown in Table 1 extracted from the arrangement feature template in Table 3, the feature function would take the following value:

[0088] If(y=='1-3-5'&&x1=='4')return 1else return 0;

[0089] If(y=='1-3-5'&&x1=='6')return 1else return 0;

[0090] If(y=='1-3-5'&&x1=='2')return 1else return 0;

[0091] If(y=='1-3-5'&&x1=='4')return 1else return 0;

[0092] If(y=='1-3-5'&&x1=='3')return 1else return 0;

[0093] If(y=='1-3-5'&&x1=='3')return 1else return 0;

[0094] If(y=='1-3-5'&&x1=='1')return 1else return 0;

[0095] If(y=='1-3-5'&&x1=='5') return 1else return 0.

[0096] For chord prediction tasks, considering the significant relationship between the current chord generation and the previously arranged chords, the designed prediction feature template needs to be more complex than the arrangement feature template. This prediction feature template includes a univariate melody feature template, a univariate chord feature template, and a univariate measure feature template. The arrangement feature template design is shown in Table 4.

[0097] Table 4

[0098]

[0099]

[0100] In this embodiment, feature extraction is performed on the input data according to the prediction feature template. The extracted prediction features are then input into the chord prediction model. The extraction process includes: taking the current beat of the melody note as the starting point, the longest-lasting note within the melody for 1, 2, 3, ..., 8 beats forward is used as the extracted melody feature; taking the current beat of the melody note as the starting point, the chords corresponding to the melody for 1, 2, 3, ..., 8 beats forward are used as the extracted chord features; and taking the measure number corresponding to the melody note in this beat as the measure feature, the 8 melody features, 8 chord features, and measure features constitute the prediction feature. Specifically, for this prediction feature template, the extracted longest note, chord, and measure features are transformed into binary functions for feature extraction. When the feature function takes a specific value, the prediction feature template is instantiated. For example, taking the training data shown in Table 2 extracted from the arrangement feature template in Table 3 as an example, the feature function will take the following value:

[0101] If(y=='4-6-1'&&x1=='4')return 1else return 0;

[0102] If(y=='4-6-1'&&x1=='4')return 1else return 0;

[0103] If(y=='4-6-1'&&x1=='6')return 1else return 0;

[0104] If(y=='4-6-1'&&x1=='2')return 1else return 0;

[0105] If(y=='4-6-1'&&x1=='4')return 1else return 0;

[0106] If(y=='4-6-1'&&x1=='3')return 1else return 0;

[0107] If(y=='4-6-1'&&x1=='3')return 1else return 0;

[0108] If(y=='4-6-1'&&x1=='1')return 1else return 0;

[0109] If(y=='4-6-1'&&x2=='4-6-1')return 1else return 0;

[0110] If(y=='4-6-1'&&x2=='4-6-1')return 1else return 0;

[0111] If(y=='4-6-1'&&x2=='4-6-1')return 1else return 0;

[0112] If(y=='4-6-1'&&x2=='2-4-6')return 1else return 0;

[0113] If(y=='4-6-1'&&x2=='2-4-6')return 1else return 0;

[0114] If(y=='4-6-1'&&x2=='6-1-3')return 1else return 0;

[0115] If(y=='4-6-1'&&x2=='6-1-3')return 1else return 0;

[0116] If(y=='4-6-1'&&x2=='6-1-3')return 1else return 0;

[0117] If(y=='4-6-1'&&x3=='III') return 1else return 0.

[0118] Specifically, the flowchart for feature extraction based on the configured feature template and the predicted feature template is as follows: Figure 6 As shown. The feature set extracted from the training data generally has a very large number of features, but not all features are effective. Too many features can reduce the efficiency of the system. To obtain representative features with less redundancy, feature filtering is necessary. Here, the common and easy-to-use threshold method is chosen as the filtering method. It uses the frequency of a feature's occurrence as the deletion criterion. If the frequency of a feature is less than the set threshold, the feature is deleted; otherwise, it is retained. The threshold is set to 3.

[0119] In this embodiment, the CRF++ 0.58 tool can be used for model training. Since the CRF model is a probabilistic statistical model in machine learning, it does not involve a large number of matrix operations compared to deep learning models. Therefore, this tool does not have a GPU training mode and can only be trained using the CPU. CPU training requires a large amount of memory to cache data, and to increase training speed, the number of threads needs to be increased as much as possible. Because this model aims to preserve chord diversity, chords are represented using pitch representation instead of chord symbols, resulting in an increase in the number of chords from the basic 432 to over 1000, further increasing the required memory. Therefore, a normal workstation cannot meet this requirement. Considering all these factors, a cloud server, ecs.r6.26xlarge, with a 104-core CPU and 768GB of memory, can be used for model training. During training, the CRF++ tool uses a feature generation algorithm based on feature templates to extract features from the training data to derive a probability matrix. Then, it updates the parameters in the probability matrix according to its internal training algorithm until convergence, ultimately producing the probability matrix model. Training the chord arrangement model on this cloud server took 15 hours, while training the chord prediction model took 20 hours. During training, the probability matrix parameters were optimized using the L-BFGS algorithm.

[0120] After training, real-time chord generation validation was performed on both models. During model prediction, since the chord arrangement and prediction models did not use binary features (i.e., they lacked a state transition matrix), a simplified form of the Viterbi algorithm could be used for chord prediction generation; the probabilities were simply calculated by adding the feature vector values.

[0121] For the chord arrangement model, during chord arrangement, the identified melodic notes of the first 8 beats are input according to the arrangement feature template. Then, the probability value of each chord is obtained by statistically analyzing the probability values ​​of the feature matrix corresponding to each output chord. The chord with the highest output probability value is used as the arrangement chord for the previous beat and input into a queue with a capacity of 4. The chord content in the queue is part of the input of Model 2. After converting the validation set to the same form as the training set, the model is validated. The obtained chord arrangement accuracy is as high as 90.4%, and the chord arrangement time per beat is less than 0.0021s. This accuracy and computational efficiency can meet the requirements of real-time chord generation and arrangement tasks.

[0122] For chord prediction models, chord prediction requires sequentially reading the melodic notes of the first 8 beats, the features of the current measure, and the arranged chords of the first 8 beats output by the chord arrangement model, based on the prediction feature template. Then, the probability values ​​of the feature matrices corresponding to each possible output chord in the current beat are statistically analyzed to obtain the probability value of each chord's occurrence. The specific process is as follows: Figure 7As shown, after converting the validation set to the same format as the training set and then validating the model, the chord prediction accuracy reached 84.6%, with a prediction time of less than 0.0025 seconds per measure. This accuracy and computational efficiency meet the requirements of real-time chord generation prediction tasks.

[0123] For the accompaniment generation unit

[0124] The accompaniment generation unit automatically matches a playing mode to the predicted chords based on the pitch change rate and cadence detection results, and then generates an accompaniment based on the predicted chords and the matched playing mode. Within the accompaniment generation unit, the system automatically switches between the input melody and the generated chords in real time, according to the set accompaniment playing mode, characteristics of the input melody, and characteristics of the generated chords, transforming the generated chords into a full accompaniment for playback, thus enhancing the music's pleasantness and richness.

[0125] In this embodiment, based on the set accompaniment playing mode, the playback order and duration of the root note, third note, fifth note, seventh note, and ninth note are automatically assigned to the generated chords. The chord playing mode can be described in the following format: [the nth note of the chord components from low to high, start time (beat), duration (beat), instrument name]. A simplified example is as follows: [1,0,1,Piano], [2,1,1,Piano], [3,2,1,Piano], [2,3,1,Piano]. This simplified example indicates that the 1st, 2nd, 3rd, and 2nd notes of the chord are played sequentially on the piano, each note lasting one beat. For example, when the components are 36, 40, 43 (C3, E3, G3), the actual performance result of this example can be visualized through the following MIDI rollout window, such as... Figure 8 As shown.

[0126] Following this playing mode format, the internal playing modes of the device were set up. The following points should be noted: 1. The playing mode is primarily designed for chords with 3-5 constituent notes, but it is compatible with chords containing more than 5 constituent notes. 2. The second and third notes of the piano accompaniment chords are raised by one octave. For guitar playing, the second note of the chord is raised by one octave in the verse, and the first and second notes are raised by one octave in the chorus. This setting is achieved internally by changing the constituent notes. For example, when the constituent notes are 36, 40, 43 (C3, E3, G3), to raise the second note of the chord by one octave, the program will change the constituent notes to 36, 43, 52, and use the updated constituent note order for the corresponding instrument. 3. If the 4th and 5th constituent notes are not present, they will not be played.

[0127] In the examples, the playing modes and their corresponding detailed formats are given as follows:

[0128] Piano 1 (verse): This is the most soothing playing mode. In the chords, the bass part is played as half notes and the treble part as quarter notes. The specific format is as follows: [1,0,2,Piano],[2,0,1,Piano],[3,0,1,Piano],[4,0,1,Piano],[5,0,1,Piano],[2,1,1,Piano],[3,1,1,Piano],[4,1,1,Piano],[5,1,1,Piano],[1,2,2,Piano],[2,2,1,Piano],[3,2,1,Piano],[4,2,1,Piano],[5,2,1,Piano],[2,3,1,Piano],[3,3,1,Piano],[4,3,1,Piano],[5,3,1,Piano].

[0129] Verse Piano 2: Imitating the electric bass playing style in pop music. Building upon Verse Piano 1, add an eighth note (the lowest note of the chord) on the last half beat of each measure, in the following format: [1,0,2,Piano],[2,0,1,Piano],[3,0,1,Piano],[4,0,1,Piano],[5,0,1,Piano],[2,1,1,Piano],[3,1,1,Piano],[4,1,1,Piano],[5,1,1,Piano] ],[1,1.5,0.5,Piano],[1,2,2,Piano],[2,2,1,Piano],[3,2,1,Piano],[4,2,1,Piano],[5,2,1 ,Piano],[2,3,1,Piano],[3,3,1,Piano],[4,3,1,Piano],[5,3,1,Piano],[1,3.5,0.5,Piano].

[0130] Verse Piano 3: Used when the verse is too long. It is designed so that the second note of the first, second, and fourth beats in a measure starts on the second half of the beat. The specific format is as follows: [1,0,2,Piano],[2,0.5,1,Piano],[3,0,1,Piano],[4,0,1,Piano],[5,0,1,Piano],[2,1.5,0.5,Piano],[3,1,1,Piano],[4,1,1,Piano],[5,1,1,Pia no],[1,2,2,Piano],[2,2,1.5,Piano],[3,2,1,Piano],[4,2,1,Piano],[5,2,1,Piano],[2,3.5,0.5,Piano],[3,3,1,Piano],[4,3,1,Piano],[5,3,1,Piano].

[0131] Piano embellishment in the verse: used at the transition of verse phrases, specifically, on the basis of piano 2 in the verse, the last beat is replaced by a sixteenth note playing the 1232 notes an octave higher than the current chord. The specific format is as follows: [1,0,2,Piano],[2,0,1,Piano],[3,0,1,Piano],[4,0,1,Piano],[5,0,1,Piano],[2,1,1,Piano],[3,1,1,Piano],[4,1,1,Piano],[5,1,1,Piano],[1,1.5,0.5,Pi ano],[1,2,2,Piano],[2,2,1,Piano],[3,2,1,Piano],[4,2,1,Piano],[5,2,1,Piano ],[1,3,1,Piano],[2,3.25,0.75,Piano],[3,3.5,0.5,Piano],[2,3.75,0.25,Piano].

[0132] Guitar verse: To reflect the progression of emotions, the guitar is introduced starting from the second verse. The specific format is as follows: [2,0.5,1,Guitar],[3,1,1,Guitar],[4,1,1,Guitar],[5,1,1,Guitar],[2,1.5,0.5,Guitar],[2,2.5,1.5,Guitar],[3,3,1,Guitar],[4,3,1,Guitar],[5,3,1,Guitar].

[0133] Unlike the verse playing mode, the chorus playing mode has different modes for each measure, so the chorus playing mode will be listed separately for each measure below.

[0134] The piano section of the chorus is designed as follows: the second note of the first and second beats of each measure changes to the second half of the beat; a root note is added to the last quarter of the second beat of each measure; the second note of the current chord is added to the second half of the third beat of each measure; all sixteenth notes of the current chord except the root note are added to the last quarter of the third beat of each measure; the first sixteenth note of the last beat of each measure is left blank, and the next three are played in sequence as 3, 2, and 1 an octave higher than the current chord; the order of the three notes on the last beat of the fourth measure is changed to 1, 2, and 3 (Cmaj7 is CEG). The specific format for each section is as follows: Section 1: [1,0,1.75,Piano],[2,0.5,1,Piano],[3,0,1,Piano],[4,0,1,Piano],[5,0,1,Piano],[2,1.5,0.5,Piano],[3,1,1,Piano],[4,1,1,Piano],[5,1,1,Piano],[1,1.75,0.25,Piano],[1,2,2,Piano],[2,2,0.5,Piano ],[2,2.5,0.25,Piano],[2,2.75,1.25,Piano],[3,2,0.75,Piano],[3,2.75,1.25,Piano],[4,2,0.75,Piano],[4,2 .75,1.25,Piano],[5,2,0.75,Piano],[5,2.75,1.25,Piano],[3,3.25,0.75,Piano],[2,3.5,0.5,Piano],[1,3.75, 0.25,Piano]; second section: [1,0,2,Piano],[2,0.5,1,Piano],[3,0,1,Piano],[4,0,1,Piano],[5,0,1,Piano],[2,1.5,0.5 ,Piano],[3,1,1,Piano],[4,1,1,Piano],[5,1,1,Piano],[1,1.75,0.25,Piano],[1,2,2,Piano],[2,2,0.5,Piano] ,[2,2.5,0.25,Piano],[2,2.75,1.25,Piano],[3,2,0.75,Piano],[3,2.75,1.25,Piano],[4,2,0.75,Piano],[4,2. 75,1.25,Piano],[5,2,0.75,Piano],[5,2.75,1.25,Piano],[3,3.25,0.75,Piano],[2,3.5,0.5,Piano],[1,3.75,0.25,Piano]; Third subsection: [1,0,2,Piano], [2,0.5,1,Piano], [3,0,1,Piano], [4,0,1,Piano], [5,0,1,Piano], [2,1.5,0.5,Piano], [3,1,1,Piano], [4,1,1,Piano], [5,1,1,Piano], [1,1.75,0.75,Piano], [1,2,2,Piano], [2,2,0.5,Piano], [2, 2.5,0.25,Piano],[2,2.75,0.25,Piano],[3,2,0.75,Piano],[3,2.75,0.25,Piano],[4,2,0.75,Piano],[4,2.75,0. 25,Piano],[5,2,0.75,Piano],[5,2.75,0.25,Piano],[3,3.25,0.75,Piano],[2,3.5,0.5,Piano],[1,3.75,0.25,Pi ano];Fourth subsection: [1,0,2,Piano], [2,0.5,1,Piano], [3,0,1,Piano], [4,0,1,Piano], [5,0,1,Piano], [2,1.5,0.5,Piano], [3,1,1,Piano], [4,1,1,Piano], [5,1,1,Piano], [1,1.75,0.25,Piano], [1,2,2,Piano], [2,2,0.5,Piano], [2,2.5,0 .25,Piano],[2,2.75,1.25,Piano],[3,2,0.75,Piano],[3,2.75,1.25,Piano],[4,2,0.75,Piano],[4,2.75,1.25,Pi ano],[5,2,0.75,Piano],[5,2.75,1.25,Piano],[1,3.25,0.75,Piano],[2,3.5,0.5,Piano],[3,3.75,0.25,Piano]. .

[0135] Sub-up song harp 2: This is a variation of sub-up song kin 1, with a little more information. Each petal concrete formalism below: First petty: [1,0,2,Piano],[2,0.75,0.75,Piano],[3,0,1,Piano],[4,0,1,Piano],[5,0,1,Piano],[2,1.5,0.25,Piano],[3,1,0.7 5,Piano],[4,1,0.75,Piano],[5,1,0.75,Piano],[2,1.75,0.25,Piano],[3,1.75,0.25,Piano],[4,1.75,0.25,Piano],[5,1.75,0.25,Pi ano],[1,2,2,Piano],[2,2.25,0.25,Piano],[2,2.5,0.25,Piano],[2,2.75,1.25,Piano],[3,2.25,0.5,Piano],[3,2.75,1.25,Piano],[ 4,2.25,0.5,Piano],[4,2.75,1.25,Piano],[5,2.25,0.5,Piano],[5,2.75,1.25,Piano],[3,3.25,0.75,Piano],[2,3.5,0.5,Piano],[1, 3.75,0.25,Piano]; Second subsection: [1,0,2,Piano], [2,0.5,1,Piano], [3,0,1,Piano], [4,0,1,Piano], [5,0,1,Piano], [2,1.5,0.25,Piano], [3,1,0.75,Piano], [4,1,0.75,Piano], [5,1,0.75,Piano], [2,1.75,0.25,Piano], [3,1.75,0.25,Piano], [4,1.75,0.25,Piano], [5,1.75,0.25 ,Piano],[1,2,2,Piano],[2,2,0.5,Piano],[2,2.5,0.25,Piano],[2,2.75,1.25,Piano],[3,2,0.75,Piano],[3,2.75,1.25,Piano],[4,2 ,0.75,Piano],[4,2.75,1.25,Piano],[5,2,0.75,Piano],[5,2.75,1.25,Piano],[3,3.25,0.75,Piano],[2,3.5,0.5,Piano],[1,3.75,0.25,Piano]; Third subsection: [1,0,2,Piano], [2,0.75,0.75,Piano], [3,0,1,Piano], [4,0,1,Piano], [5,0,1,Piano], [2,1.5,0.25,Piano], [3,1,0.75,Piano], [4,1,0.75,Piano], [5,1,0.75,Piano], [2,1.75,0.25,Piano], [3,1.75,0.25,Piano], [4,1.75,0.25,Piano], [5,1.75,0.25, Piano],[1,2,2,Piano],[2,2.25,0.25,Piano],[2,2.75,1.25,Piano],[3,2.25,0.25,Piano],[3,2.75,1.25,Piano],[4,2.25,0.25,Piano no],[4,2.75,1.25,Piano],[5,2.25,0.25,Piano],[5,2.75,1.25,Piano],[3,3.25,0.75,Piano],[2,3.5,0.5,Piano],[1,3.75,0.25,Piano no];Fourth subsection: [1,0,2,Piano], [2,0.5,1,Piano], [3,0,1,Piano], [4,0,1,Piano], [5,0,1,Piano], [2,1.5,0.25,Piano], [3,1,0.75,Piano], [4,1,0.75,Piano], [5,1,0.75,Piano], [2,1.75,0.25,Piano], [3,1.75,0.25,Piano], [4,1.75,0.25,Piano], [5,1.75,0.25,Piano], [1, 2,2,Piano],[2,2,0.5,Piano],[2,2.5,0.25,Piano],[2,2.75,1.25,Piano],[3,2,0.75,Piano],[3,2.75,1.25,Piano],[4,2,0.75,Piano o],[4,2.75,1.25,Piano],[5,2,0.75,Piano],[5,2.75,1.25,Piano],[1,3.25,0.75,Piano],[2,3.5,0.5,Piano],[3,3.75,0.25,Piano]. .

[0136] Piano embellishment in the chorus: This is used when embellishing the chorus and consists of a long note for one measure. The specific format is as follows: [1,0,4,Piano],[2,0,4,Piano],[3,0,4,Piano],[4,0,4,Piano],[5,0,4,Piano].

[0137] Guitar chorus: Designed based only on the first three notes of the chords. The specific format of each measure is as follows: Measure 1: [1,0.5,1.25,Guitar],[2,0.75,1.25,Guitar],[3,1.25,0.75,Guitar],[1,1.75,0.25,Guitar],[1,2,1.75,Guitar],[3,2,0.75,Guitar],[2,2.5,0.75,Guitar],[3,2.75,1.25,Guitar],[2,3.25,0.75,Guitar] r],[1,3.75,0.25,Guitar];Second section: [1,0,1.75,Guitar],[2,0.75,1.25,Guitar],[3,1.25,0.75,Guitar],[1,1.75,0.25,Guitar],[2,2,0.75,Guitar],[1,2.5,1.5,Guitar],[2,2.75,1.25,Guitar],[3,3.25,0.75,Guitar];Third section :[2,0,0.75,Guitar],[1,0.5,1.25,Guitar],[2,0.75,1.25,Guitar],[3,1.25,0.75,Guitar],[1,1.75,0.25,Gu itar],[1,2.25,1.5,Guitar],[2,2.5,1,Guitar],[3,2.75,1.25,Guitar],[2,3.25,0.75,Guitar],[1,3.75,0.25 ,Guitar]; Section 4: [1,0,1.75,Guitar],[2,0.75,1.25,Guitar],[3,1.25,0.75,Guitar],[1,1.75,0.25,Guitar],[ 1,2,1.75,Guitar],[2,2.5,1,Guitar],[3,2.75,1.25,Guitar],[2,3.25,0.75,Guitar],[1,3.75,0.25,Guitar].

[0138] When generating the accompaniment, the above playing patterns are converted into a digital format that is easy for the processor to understand, and executed line by line. This format has a one-to-one relationship with the actual MIDI instructions, uniquely writing MIDI instructions as text. Examples of text in this format are as follows: 0:0->1|0,0=>1|0, 16:0<=1|0,1=>1|0, 16:1<=1|0,2=>1|0, 16:2<=1|0,1=>1|0. The specific meaning is as follows: For example, looking from left to right, 0:0->1|0, 0: means wait 0 / 16 beats before executing this line, -> means set the instrument used by the track, so 0->1 means set track 1 to instrument 0 (i.e., piano; other meanings can be found in the MIDI instrument table), |0 means wait 0 milliseconds before executing this statement, => means play the specified note, so 0=>1 means play the 0th note on track 1; for the next 16:0<=1|0,1=>1|0, wait 16 / 16 beats (i.e., 1 beat), <= means stop playing the specified note, so 0<=1 means stop playing the 0th note on track 1.

[0139] The text in the above format is parsed into MIDI instructions by the program. The MIDI instructions can be output as MIDI format files through midiffile, or they can be played directly by calling the Tiny Sound Font sound library.

[0140] In this embodiment, the playing style is preset internally by the developers. For example... Figure 9 As shown, during the initial stage of the improvisational accompaniment device, the playing mode is automatically matched to the predicted chord based on the pitch change rate. Specifically, the pitch change rate is determined by calculating the absolute value of the pitch difference between the current note and the previous note based on the input chord. This pitch change rate then determines whether the current musical phrase belongs to the verse or chorus type, switching between verse and chorus playing modes accordingly. When it is a verse type, it plays according to several preset verse playing modes for chord matching; when it is a chorus type, it plays according to several preset chorus playing modes for chord matching.

[0141] In this embodiment, generating accompaniment based on predicted chords and matching playing patterns includes: playing the predicted chords in matching playing patterns to generate accompaniment; when it is a verse type, multiple matching verse regular playing patterns are played alternately; when it is a chorus type, multiple matching chorus regular playing patterns are played alternately; playing between the verse and chorus in a verse embellishment pattern or a chorus embellishment pattern; detecting and determining whether a musical phrase has ended by chord cadences; when the end of a musical phrase is detected, playing in an embellishment pattern at the phrase as a transition between musical phrases.

[0142] Specifically, as shown in the embodiment, chord cadence detection is used to determine whether a musical phrase or section has ended. When the end of a phrase or section is detected, the system will play a one-bar embellished accompaniment at the transition point as a transition between phrases; when a phrase is playing normally, the system will play the regular playing mode, and after playing four bars of the same regular playing mode, it will switch to the next regular playing mode. The embodiment switches every four bars because, in general, the length of a song section is often a positive integer multiple of four bars. The specific switching method is as follows: Since most songs begin with a verse, the applicant will initially play the verse piano 1 and verse piano 2 in sequence, and only after these are played will a check be performed every four bars to determine whether the chorus has begun. If the song enters the chorus, it will be played in a loop in the order of chorus piano 1 and chorus guitar, then chorus piano 2 and chorus guitar; if the song enters the verse, it will be played in a loop in the order of verse piano 1 and verse guitar, then verse piano 2 and verse guitar, then verse piano 3 and verse guitar.

[0143] In this embodiment, the determination of whether a musical phrase has ended is made by detecting chord cadences, including: starting with the chord of the current beat, tracing back to the chords of the previous K beats; if a cadence / half-cadence chord progression can be formed, then chord phrasing is performed, and the musical phrase is considered to have ended. For example, for each generated beat of chord, the preceding 4 measures (16 beats) are traced back to see if a cadence / half-cadence chord progression can be formed within it; if so, chord phrasing is performed. Cadence / half-cadence chord progressions refer to harmonic progressions used to end a piece of music, and are mainly divided into: 1. authentic cadence, i.e., the cadence progression of VI; 2. apocryphal cadence, i.e., the cadence progression of IV-I, II-I, VI-I; 3. obstructive cadence, i.e., the progression originally V7-I is changed to V7-VI.

[0144] Figure 10 Example diagrams of some piano playing patterns in staff notation are provided to help reviewers understand the changing trends in texture. For example... Figure 10 As shown, 1 and 2 are piano 1 and piano 2 for the verse, respectively, and 3 to 6 are piano 1 for the chorus. The rhythm and auditory design correspond to the emotional changes in the verse and chorus.

[0145] During the accompaniment generation process, a metronome (e.g., drum sounds) needs to be played according to the beat intervals and tempo. Specifically, the drum sound playback process is as follows: Figure 11 As shown.

[0146] In this embodiment, the melody detection unit, chord arrangement unit, chord prediction unit, and accompaniment generation unit are all implemented on a motherboard with an ARM architecture chip. This chip controls the overall process of the melody detection unit, chord arrangement unit, chord prediction unit, and accompaniment generation unit, and only needs to support the Android 4.0 or higher operating environment.

[0147] Reverb unit

[0148] like Figure 1 As shown, the improvisational accompaniment generation device provided in this embodiment also includes a reverb unit, used to perform noise reduction processing on the audio data collected from human voices. The noise-reduced audio data is then input to a melody detection unit for melody detection. The reverb unit also mixes the accompaniment generated by the accompaniment generation unit with the noise-reduced audio data and outputs the result. Specifically, the reverb unit uses a reverb DSP chip: it performs noise reduction and reverb operations on the human voice input from the microphone to obtain wet sound (referring to pure human voice that does not contain the music itself and has been processed) and outputs it to the Android module for processing. Finally, it mixes the wet sound with the accompaniment generated by the Android motherboard.

[0149] Human voice acquisition unit

[0150] like Figure 1 As shown, the improvisational accompaniment generation device provided in this embodiment also includes a voice acquisition unit for acquiring audio data of human voices. The voice acquisition unit can use a microphone to acquire the audio data of human voices.

[0151] Playback Unit

[0152] like Figure 1 As shown, the improvisational accompaniment generation device provided in this embodiment also includes a playback unit for playing the audio data with accompaniment output by the reverberation unit. In this embodiment, the playback unit can be implemented using a speaker.

[0153] Visual editing unit

[0154] like Figure 1 As shown, the improvisational accompaniment generation device provided in this embodiment also includes a visual editing unit, which controls the start or stop of the accompaniment and configures parameters through a visual interface. These parameters include tonality, metronome speed (beat rate), zero-crossing rate, minimum recognition volume, whether the metronome is on and the metronome interval, whether to filter all semitones, whether to filter the starting semitone, chord harmony, and chord constraints.

[0155] Specifically, for filtering all semitones, if the note name corresponding to a chord contains the semitone marker "#", the chord is filtered out; for filtering the starting semitone, if the note name corresponding to the root note (i.e. the first note) of a chord contains the semitone marker "#", the chord is filtered out.

[0156] For the calculation of chord harmony, firstly, the absolute value of the interval difference between two notes is calculated and modulo 12 is performed. According to the above value, the harmony between the two notes is as follows: note_harmony

[12] ={144,11,20,27,32,35,36,35,32,27,20,11}; then, according to note_harmony

[12] , the harmony between each note in the chord and the current melody note is obtained, accumulated and averaged. Only when the average harmony of the chord reaches the set threshold can it be played as an accompaniment chord.

[0157] Regarding chord constraints, the calculation method for the matching degree between the generated accompaniment chords and the corresponding chord templates includes: comparing all the system's built-in chord templates (such as 1 5 6 3 4 1 2 5 or 4 5 3 6 2 5 1) with the chords in the current playback period, calculating their absolute interval differences, and calculating the average harmony of all templates according to the above method. The one with the largest average harmony among these templates is selected, and only the one with the largest average harmony that reaches the set chord constraint threshold can be played as an accompaniment chord.

[0158] In summary, the application process of the improvisational accompaniment generation device provided in the embodiments includes: as follows Figure 12 As shown, after setting various system parameters through the on-screen human-computer interaction interface, clicking the start button in the device will automatically input the human voice data through the microphone after noise reduction by the reverb module. Once a full beat of data has been collected, its fundamental tone is extracted and converted into a scale, and the melody in the corresponding key is derived. Then, using the extracted melody as the melody for the previous beat, the input chord arrangement model is used to arrange the chords for the previous beat. The arranged chords are then used to predict the chords using the input chord prediction model. After generation, the program will generate accompaniment in real time according to the accompaniment playing style set within the system. Simultaneously, during operation, the program will distinguish between the verse and chorus based on the speed of pitch change and switch the accompaniment playing style based on the cadence detection results, providing the user with a better auditory experience.

[0159] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An improvisational accompaniment generation device, characterized in that, include: A melody detection unit is used to perform melody detection on the acquired audio data to obtain a melody, wherein the melody is divided by beat; The chord arrangement unit is used to perform chord arrangement based on the chord arrangement model built on CRF, according to the melody notes of multiple consecutive beats, so as to obtain the arranged chords corresponding to the melody notes of multiple beats and store them as a queue; wherein, for each input melody note, according to the current total number of beats N, the melody note with the longest duration in the Nth, N-1th, N-2th, ..., NM-1th beats is taken as the extracted melody feature, so as to obtain the arrangement feature composed of M melody features, and the extracted arrangement feature is input into the chord arrangement model; The chord prediction unit is used to predict chords in real time using a chord prediction model built on CRF, based on the melody notes of multiple consecutive beats, the chord arrangements corresponding to the melody notes of multiple beats, and the measure number corresponding to the melody note of the current beat. Specifically, for each input data, based on the current total number of beats N, the longest-lasting note in the melody of beats N-1, N-2, N-3, ..., NM is taken as the extracted melody feature, the chord arrangements corresponding to the melody of beats N-1, N-2, ..., NM are taken as the extracted chord feature, and the measure number corresponding to the melody note of the current beat is taken as the measure feature. The M melody features, M chord features, and measure features are combined to form the prediction features and input into the chord prediction model. The accompaniment generation unit is used to automatically match the playing pattern for the predicted chords based on the pitch change rate and cadence detection results, and then generate an accompaniment based on the predicted chords and the matched playing pattern.

2. The improvisational accompaniment generation device according to claim 1, characterized in that, For the chord arrangement model, the value of M is 8.

3. The improvisational accompaniment generation device according to claim 1, characterized in that, For the chord prediction model, the value of M is 8.

4. The improvisational accompaniment generation device according to claim 1, characterized in that, Before the chord arrangement model and the chord prediction model are applied, supervised learning using training samples is required to optimize the model parameters. The process of constructing training samples includes: cleaning the acquired music dataset, extracting and sampling melody and chord data, shifting the sampled data by an octave, and unifying the tonality. The extraction and sampling of melody and chord data includes: converting the cleaned music file into a MIDI file of a melody track; extracting the main melody track data from the MIDI file and converting it into two-dimensional data including a timing array and a pitch array; sampling the pitch according to the timing information to extract the melody scale information and obtain a one-dimensional timing array of the melody. The data with chord markings are extracted separately from the cleaned music files, and the chord pitch information at the same time point is converted into component notes in MIDI pitch form. The chords are stored according to the component notes, resulting in two-dimensional data including the component note pitch array and timing data. The chord information of the component notes is sampled according to the timing information to obtain a one-dimensional timing array of the chord component notes.

5. The improvisational accompaniment generation device according to claim 1, characterized in that, The accompaniment generation unit automatically matches the playing mode for the predicted chords based on the pitch change rate and cadence detection results, including: The absolute value of the pitch difference between the current note and the previous note is calculated based on the input chord to determine the pitch change rate. The pitch change rate is then used to determine whether the current musical phrase belongs to the verse or chorus type, so as to switch between the verse playing mode and the chorus playing mode. When it is a verse type, there are multiple preset verse playing modes for chord matching; When it is a chorus type, there are multiple preset chorus playing modes for chorus matching; By detecting chord cadences and determining whether a musical phrase has ended, when a musical phrase is detected in progress, if it is a verse type, multiple matching verse regular playing patterns are played alternately; if it is a chorus type, multiple matching chorus regular playing patterns are played alternately. When the end of a musical phrase is detected, the phrase will be played in either the verse embellishment mode or the chorus embellishment mode as a transition between phrases.

6. The improvisational accompaniment generation device according to claim 5, characterized in that, The accompaniment generation unit detects and determines whether a musical phrase has ended by checking for chord cadences, including: Starting with the chord of the current beat, trace back to the chords of the previous K beats. If a cadence / half-cadence chord progression can be formed, then chord phrasing is performed, and the musical phrase is considered to have ended.

7. The improvisational accompaniment generation device according to claim 1, characterized in that, The improvisational accompaniment generation device also includes a reverberation unit, which is used to reduce the noise of the audio data collected from human voices. The noise-reduced audio data is then input to the melody detection unit for melody detection. It is also used to mix and output the accompaniment generated by the accompaniment generation unit with the noise-reduced audio data.

8. The improvisational accompaniment generation device according to claim 7, characterized in that, The improvisational accompaniment generation device also includes: The human voice acquisition unit is used to acquire audio data of human voices; The playback unit is used to play the audio data with accompaniment output by the reverb unit; The visual editing unit allows users to control the start or stop of the accompaniment and configure parameters through a visual interface. These parameters include: key, metronome speed, zero-crossing rate, minimum recognition volume, whether to turn on the metronome and metronome interval, whether to filter all semitones, whether to filter the starting semitone, chord harmony, and chord constraints.

Images