Performance information prediction device, performance model training device, performance information generation system, performance information prediction method, and performance model training method
The system uses spectral data frames and trained models to accurately predict guitar playing techniques and note numbers, addressing tracking errors and misinterpretations in guitar controllers.
Patent Information
- Application Number
- JP2020158761
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-09-23
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-09-23
AI Technical Summary
Existing guitar controllers struggle with tracking errors and misinterpretation of playing techniques due to transient performance noises and overtone movements, leading to inaccurate conversion of performance information.
A system that generates spectral data frames from string vibration waveforms, using a trained performance model to predict playing techniques and note numbers by analyzing spectral characterization data frames before and after a reference time, incorporating machine learning models like neural networks.
Accurately converts guitar performance into precise performance information, reducing tracking errors and improving the interpretation of playing styles and note recognition.
Smart Images

Figure 0007585681000001 
Figure 0007585681000002 
Figure 0007585681000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a performance information prediction device, a performance model training device, a performance information generating system, a performance information prediction method, and a performance model training method. [Background technology]
[0002] There is an electronic musical instrument called a guitar controller (or guitar synthesizer) that converts the string vibration waveform of an instrument such as a guitar into an electric signal using a magnetic or piezo pickup, and then analyzes the pitch and volume to convert it into digital performance data such as MIDI (Musical Instrument Digital Interface) messages. Unlike dedicated guitar controllers that are only used to play sound sources, this type of controller has the great advantage of being able to play MIDI by equipping each string with an independent pickup to obtain performance information while retaining the shape and functions of a normal guitar, and is the most common form. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Japanese Patent Application Publication No. 9-6339 [Patent Document 2] JP 2000-105590 A Summary of the Invention [Problem to be solved by the invention]
[0004] However, one of the major problems that has not been solved for many years is the so-called tracking error, where the performance information is converted into unintended performance information when the strings are plucked. This is because the pitch is detected by observing the peaks of the input signal waveform and the cycle of the zero crossing points, and the sound generation method is determined only from the amount of change in the input envelope, so it is fooled by transient performance noise caused by picking or tapping when plucking the strings, and by the complex overtone movements of the strings.
[0005] For example, there are cases where picking noise, which is the friction noise that occurs when the pick touches the string before plucking, and picking noise caused by the vibration of the very short length of the string between the pick and the bridge, are recognized as playing sounds. This can result in note information that is far higher than the playing pitch that corresponds to the fret position where the string is actually held down.
[0006] In addition, there are cases where harmonics are not generated by intentional playing, but rather the amount of harmonics contained in the string vibration during normal playing is so large that it is difficult to distinguish between the fundamental tone and the overtone, and the harmonic pitch is recognized as the played note. The most common case is that the second overtone is mistaken for a pitch one octave higher than the actual pitch, but there are also cases where the third overtone is mistaken for the fundamental tone.
[0007] Regarding the misrecognition of picking and legato playing, when the sound production technique of guitar strings is judged and performance information is added, there are cases where it is misunderstood as an unintended sound production technique. For example, the sound production techniques of guitar strings can be classified as follows based on the characteristics of the sound. A. Normal plucking with picking or fingers B. Hammer-on (using the fingers of the hand that presses the frets) or tapping (using the fingers of the opposite hand to the one that normally presses the strings) is a technique in which a string is pressed at a certain fret by hitting or touching a higher fret with another finger, changing the pitch. C. Pull-off technique: Pull the string slightly with the finger pressing the fret and then release, or use the same finger as tapping to pluck the string. D. A glissando or slide technique in which the pitch is changed by pressing and sliding a finger over the currently sounding string. This is usually not interpreted as a new sound being produced by plucking the string in a MIDI message, but is instead expressed as a pitch change.
[0008] Generally, in cases A, B, and C, new sounding information is generated, and in case D, it is determined to be legato playing, and pitch bend information is generated for the currently sounding note.
[0009] It is believed that these playing styles can be determined not only by analyzing changes in the volume envelope when the sound is produced, but also by analyzing changes in the levels of the harmonics of various noises that occur during transients. However, this type of analysis has been difficult using conventional methods.
[0010] Furthermore, the frequency components and changes when plucking the strings can vary greatly depending on the different characteristics of each guitar, the player's habits, the shape and material of the pick, and (in fingerpicking) the hardness of the player's skin, so it is necessary to take these individual characteristics into consideration when making a judgment.However, the reality is that there are no instruments that take such factors into account when making a judgment, and in fact there are almost no instruments that can judge playing style itself.
[0011] In view of the above problems, an object of the present disclosure is to provide a technology for converting the performance of an electronic stringed instrument into performance information with high accuracy. [Means for solving the problem]
[0012] In order to solve the above problem, one aspect of the present disclosure is a method for generating a spectrum data frame from string vibration waveform data representing a stringed instrument performance, and performing a calculation based on the spectrum data frame: A frequency band consisting of a predetermined number of upper peaks included in the spectrum data frame The preprocessing section obtains the spectral feature data frame, and the trained performance model is used to obtain the reference time. The aboveA spectral characterization data frame and the reference time The above before and after the time A performance information prediction unit that predicts performance information of the stringed instrument performance from the spectral feature data frame. the trained performance model obtains the spectral feature data frame at the reference time, a first number of spectral feature data frames before the reference time, and a second number of spectral feature data frames after the reference time, and outputs a playing style and a note number of a stringed instrument. The present invention relates to a performance information prediction device. Effect of the Invention
[0013] According to the present disclosure, the performance of an electronic stringed instrument can be converted into performance information with high accuracy. [Brief description of the drawings]
[0014] [Figure 1] FIG. 1 is a schematic diagram illustrating a guitar controller according to one embodiment of the present disclosure. [Diagram 2] A diagram showing the configuration of performance information according to one embodiment of the present disclosure. [Diagram 3] FIG. 1 illustrates a tablature according to an embodiment of the present disclosure. [Figure 4] FIG. 1 illustrates an external view of a guitar controller according to an embodiment of the present disclosure. [Diagram 5] FIG. 2 is a block diagram showing a hardware configuration of a guitar according to an embodiment of the present disclosure. [Figure 6] FIG. 2 is a block diagram showing a hardware configuration of a control device according to an embodiment of the present disclosure. [Figure 7] 1 is a schematic diagram illustrating an operation of a musical performance model training device according to an embodiment of the present disclosure. [Figure 8] 1 is a block diagram showing a functional configuration of a musical performance model training device according to an embodiment of the present disclosure. [Figure 9] FIG. 2 is a schematic diagram illustrating a spectral data frame according to an embodiment of the present disclosure. [Figure 10] FIG. 2 is a schematic diagram illustrating a characterization data frame according to an embodiment of the present disclosure. [Figure 11] A diagram showing the architecture of a performance model according to one embodiment of the present disclosure. [Figure 12] A diagram showing the architecture of a performance model according to another embodiment of the present disclosure. [Figure 13] 13 is a flowchart illustrating a performance model training process according to an embodiment of the present disclosure. [Figure 14] 1 is a schematic diagram illustrating an operation of a performance information prediction device according to an embodiment of the present disclosure. [Figure 15] 1 is a block diagram showing a functional configuration of a performance information prediction device according to an embodiment of the present disclosure. [Figure 16] FIG. 2 is a schematic diagram illustrating volume detection and pitch detection according to one embodiment of the present disclosure. [Figure 17] 11 is a flowchart illustrating a performance information prediction process according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0015] In the following embodiment, a guitar controller that generates performance information (e.g., MIDI messages) from a string vibration waveform generated by playing a guitar is disclosed. Note that the present disclosure is not limited to a guitar controller, and may be applied to any other performance information generating device that generates performance information from the performance of a stringed instrument having a string vibration waveform extraction function. [Summary of this disclosure] 1, a guitar controller 10 according to an embodiment of the present disclosure includes a guitar 50 and a control device 100. The guitar controller 10 generates performance information from a string vibration waveform generated by playing the guitar 50, using a performance model realized as a machine learning model such as a neural network.
[0016] As shown in FIG. 2, the performance information according to an embodiment of the present disclosure indicates the performance type of sound generation information, sound muting information, and pitch change information.
[0017] The sound generation information indicates the note number and playing style determined by the performance model as a classification model, and the strength of the string plucking determined by envelope detection. The playing styles are classified into seven types, for example: 0) picking with a pick, 1) finger picking, 2) hammer-on (tapping), 3) pulling off, 4) mute picking, 5) open harmonics, and 6) picking harmonics. When sound generation is expressed by MIDI messages, the playing style may be represented by Control Change: 0xBn, 0x46, vv, and the note number and string plucking strength may be represented by Note On: 0x9n, kk, vv.
[0018] Moreover, the mute information is detected by envelope detection, and indicates 0) sound stop and 1) replacement. When mute is expressed by a MIDI message, it may be expressed by Control Change: 0xBn, 0x46, vv and Note Off: 0x8n, kk, vv.
[0019] The pitch change information is detected by zero cross counting, and indicates, for example, half-tone bend up, half-tone bend down, whole-tone bend up, whole-tone bend down, one-and-a-half-tone bend up, one-and-a-half-tone bend down, two-tone bend up, two-tone bend down, and slide. When expressing pitch change by MIDI message, it may be expressed by Pitch Bend: 0xEn,ll,mm.
[0020] In the embodiment shown in FIG. 1, the guitar controller 10 has two operating modes: a training mode for training a performance model, and a performance mode for predicting performance information using the trained performance model. The control device 100 has a performance model training device 200 used in the training mode, and a performance information prediction device 300 used in the performance mode.
[0021] First, in the training mode, the guitar controller 10 acquires training data from the training performance information database 80. The training data is composed of, for example, a pair of music score data (for example, TAB scores, etc.) and a MIDI file corresponding to the music score data. The TAB scores may be written according to a well-known notation, for example, as shown in FIG. 3. When the user plays the guitar 50 according to the music score of the acquired training music score data, the performance model training device 200 inputs string vibration information generated by the guitar 50 based on the user's performance to the performance model to be trained, compares the MIDI message as performance information output from the performance model with the training MIDI file, and trains the performance model so that the error between them is reduced. In the present disclosure, the string vibration waveform data is converted into spectrum data by a fast Fourier transform (FFT), and performance information is acquired from the performance model using spectrum feature data characterized based on a predetermined number of peaks in the spectrum data. When the training is completed, the performance model training device 200 provides the trained performance model to the performance information prediction device 300.
[0022] Next, in the performance mode, when the user plays the guitar 50, the performance information prediction device 300 inputs string vibration information generated by the guitar 50 based on the user's performance into the trained performance model, and acquires performance information such as MIDI messages. The acquired performance information is transmitted to, for example, an external playback device or computer, and the user can play back the performance by the user via the playback device or use the performance information on the computer.
[0023] This reduces tracking errors when converting the performance of an electronic stringed instrument into performance information, and makes it possible to determine the playing style with high accuracy.
[0024] It should be noted that the guitar controller 10 according to the embodiment described below includes the performance model training device 200, but the present disclosure is not limited thereto. For example, the performance model may be trained by an external computer or server, and the trained performance model and / or update information of the performance model may be provided to the performance information prediction device 300 from the external computer or server. [Hardware configuration] Next, the physical configuration of the guitar controller 10 will be described with reference to Fig. 4. Fig. 4 is a diagram showing the external appearance of the guitar controller 10 according to one embodiment of the present disclosure.
[0025] As shown in FIG. 4, the guitar controller 10 is a separate type performance information generating system that is composed of a guitar 50 and a control device 100 that are interconnected.
[0026] The guitar 50 is a normal electric guitar equipped with a hexa-divided pickup for picking up the independent vibrations of each of the six strings, a MIDI volume for controlling the volume of the performance information, and an up / down switch for switching the patch memory number up or down for the control device 100. This information and the output of the normal pickup are sent to the control device 100 via a multi-cable. Power is also supplied from the control device 100 via the multi-cable. The hexa-divided pickup of this embodiment is a magnetic pickup, the same as a normal pickup.
[0027] On the other hand, the control device 100 receives input of guitar string vibrations and generates performance information in MIDI format. The destination of the performance information may be, without limitation, a sound source unit, a computer, etc. As shown in FIG. 1, the control device 100 has a foot switch for switching the bank number and number of the patch memory in which various settings are stored, a CONTROL switch and a foot pedal that can assign and transmit any performance message. The number of the currently selected patch memory is displayed on the BANK / NUM screen. An LCD is used as the main display device, but a touch panel is also attached on the screen. A rotary encoder for inputting data is also provided on the panel. As terminals, the device is provided with a GUITAR INPUT input terminal for a multi-cable from the guitar 50, a GUITAR OUT audio output terminal for a normal pickup, a MIDI OUT output terminal for a MIDI performance signal, a USB to HOST connection terminal for a host computer, and an AC power input terminal AC POWER.
[0028] Next, a hardware configuration of a guitar 50 according to an embodiment of the present disclosure will be described with reference to Fig. 5. Fig. 5 is a block diagram showing a hardware configuration of a guitar 50 according to an embodiment of the present disclosure.
[0029] 5, from the guitar 50, the signal passed through the buffer amplifier of the hexa-divided pickup, the MIDI volume control, the up / down switch of the patch memory, and the signal of the normal pickup are sent to the control device 100 via a multi-cable. The three normal pickups are selected by the pickup selector, and the signal passed through the buffer amplifier via the tone control circuit and the volume control circuit is sent to the control device 100.
[0030] Next, a hardware configuration of the control device 100 according to an embodiment of the present disclosure will be described with reference to Fig. 6. Fig. 6 is a block diagram showing a hardware configuration of the control device according to an embodiment of the present disclosure.
[0031] As shown in FIG. 6, the control device 100 is composed of a CPU (Central Processing Unit) and a DSP (Digital Signal Processor). The CPU manages the overall functions and processing of the control device 100, and the DSP executes waveform analysis processing that requires high-speed processing. The CPU bus is connected to the RAM used by the CPU, the Flash ROM, the LCD controller that controls the LCD, the I / O interface connected to various I / O devices, the DSP, the USB interface, and the MIDI interface. Furthermore, the I / O interface is connected to a foot switch, a rotary encoder, an LCD touch panel, the MIDI volume of the guitar 50, an A / D converter for detecting the position of the pedal of the control device 100, and an LED for displaying the patch memory number. Although only one A / D converter is shown in the figure, the input source is switched in a time-division manner by a multiplexer to read the value. The DSP, to which a dedicated RAM and Flash ROM are connected, is connected to an independent A / D converter for quickly converting the output of the six strings of the hexa-divided pickup into a digital signal, enabling high-speed analysis processing.
[0032] However, the guitar 50 and the control device 100 are not limited to the above-mentioned hardware configuration, and may be realized by any other appropriate hardware configuration. [Performance model training device] Next, a musical performance model training device 200 according to an embodiment of the present disclosure will be described with reference to Figures 7 to 12. Figure 7 is a schematic diagram showing the operation of the musical performance model training device 200 according to an embodiment of the present disclosure.
[0033] As shown in FIG. 7, the performance model training device 200 trains the performance model using training data stored in the training performance information database 80 that stores the training data. Specifically, the training data is composed of a pair of training score data and training performance information corresponding to the score data, and a score (e.g., TAB score, etc.) displayed based on the training score data is displayed to the performer, who plays the guitar 50 under tempo control by a metronome. String vibration waveform data representing the performance is provided to the performance model training device 200, which converts the acquired string vibration waveform data into a spectral feature data frame, which will be described in detail below, and inputs the spectral feature data frames at the reference time and around the reference time to the performance model to be trained. The performance model training device 200 then compares the output from the performance model with the sound information (e.g., note number and playing style, etc.) of the training performance information, and updates the parameters of the performance model according to the difference. The performance model training device 200 repeats the above-mentioned process until a predetermined end condition is satisfied, and optimizes the performance model so that the output from the performance model approaches the sound generation information of the training performance information.
[0034] FIG. 8 is a block diagram showing a functional configuration of a musical performance model training device 200 according to an embodiment of the present disclosure.
[0035] As shown in FIG. 8, the musical performance model training device 200 includes a preprocessing unit 210 and a musical performance model training unit 220.
[0036] The pre-processing unit 210 generates a spectral data frame from string vibration waveform data representing a stringed instrument performance performed according to the training performance information, and converts the spectral data frame into a spectral feature data frame.
[0037] Specifically, when the guitar 50 is played by a player, the guitar 50 acquires string vibration waveform data indicating time and the amplitude of each string as shown in FIG. 9, and transmits the data to the performance model training device 200. That is, since the guitar 50 has six strings, six types of string vibration waveform data are generated. The preprocessing unit 210 performs a fast Fourier transform (FFT) on the string vibration waveform data of each string to acquire spectrum data. Specifically, the preprocessing unit 210 may extract string vibration waveform frames with a window width w (e.g., W=512, 25.6 msec, etc.) that overlap on the time axis from the string vibration waveform data, and perform an FFT for every I samplings (I=64, 3.2 msec, etc.), to convert each string vibration waveform frame into a spectrum data frame.
[0038] After conversion into the spectrum data frame, the preprocessing unit 210 characterizes the spectrum data frame by a predetermined number of the top peaks of each spectrum data frame. For example, when characterizing the spectrum data frame by the top four peaks, the preprocessing unit 210 characterizes the spectrum data frame by the frequencies of the top four peaks (maximum points) among the peaks (maxima) on the frequency axis in the spectrum data frame as shown in Fig. 10, and constructs a spectrum characterization data frame by the frequencies of the four peaks. This characterization is considered to reduce the data size and to emphasize the peak heights and time changes in the heights from the peaks that are assumed to be related to the playing style and note number to be predicted, thereby improving the accuracy and speeding up the performance model training process.
[0039] The pre-processing unit 210 generates spectral feature data consisting of a predetermined number of peaks extracted in this manner, and provides the data to the performance model training unit 220 .
[0040] The performance model training unit 220 uses the training performance information to train a performance model that predicts performance information of a stringed instrument performance from a spectral feature data frame at a reference time and spectral feature data frames before and after the spectral feature data frame at the reference time. Here, when predicting the playing style and note number at the reference time to be predicted, the performance model to be trained acquires not only the spectral feature data frame at the reference time but also the spectral feature data frames at times before and after the reference time as input, and outputs the playing style and note number at the reference time. For example, the performance model training unit 220 may input the spectral feature data frame at the reference time, p spectral feature data frames immediately before the reference time, and n spectral feature data frames immediately after the reference time to the performance model. Here, the predetermined numbers p and n may be the same or different predetermined values. For example, it is preferable that the predetermined numbers p and n are set to values such that the time lag between the player plucking the strings of the guitar 50 and the output of the performance information in the performance information prediction device 300 cannot be recognized by the player.
[0041] By using spectral feature data frames over a certain time range in this way, it is possible to determine whether a new pluck has occurred by taking into account the context between frames, and to determine changes in the pluck over time.
[0042] If no sound is produced at the reference time, that is, if the reference time is in a muted state, the performance model may be trained to output a value indicating that detection is not possible.
[0043] In one embodiment, the performance model may be realized by a neural network. For example, the performance model may be a neural network having a network architecture as shown in Fig. 11. In this case, the performance model training unit 220 inputs a spectral feature data frame of a reference time and (p+n) spectral feature data frames around the reference time to the input layer of the neural network, and obtains a rendition style number Var and a note number Note from the output layer through calculations in the intermediate layer.
[0044] In another embodiment, the performance model may be realized by a recurrent neural network as shown in Fig. 12. In this case, the performance model training unit 220 may use a spectral feature data frame of a wider time range than the above-mentioned time range of p, n. For example, the spectral feature data frame of the reference time t, b (b>p) spectral feature data frames immediately before the reference time, and f (f>n) spectral feature data frames immediately after the reference time are input to the input layer X t-b ,···X t-1 ,X t ,X t+1 ,···,X t+f and obtain the rendition style number Var and the note number Note from the output layer through calculations in the intermediate layer. A recurrent neural network is suitable for processing time-series data, and is considered to be able to predict the rendition style number Var and the note number Note with high accuracy.
[0045] The performance model to be trained may be a pre-trained machine learning model, and the performance model training unit 220 may fine-tune the pre-trained performance model by the above-mentioned training process. This makes it possible to build a highly accurate performance model with less training data than training a performance model from an initial machine learning model.
[0046] When the performance model training unit 220 acquires the playing style and note number from the performance model, the performance model training unit 220 compares the acquired playing style and note number with the playing style and note number of the training performance information, and updates the parameters of the performance model so that they match. For example, when the performance model is realized by a neural network, the performance model training unit 220 may update the parameters of the neural network according to the comparison result according to the well-known backpropagation method.
[0047] The performance model training unit 220 repeats the above-mentioned process to train the performance model until a predetermined termination condition is satisfied, and when the predetermined termination condition is satisfied, passes the performance model at that time point as a trained performance model to the performance information prediction device 300. Here, the predetermined termination condition may be that all prepared training data has been processed, or the like. [Performance model training process] Next, a musical performance model training process according to an embodiment of the present disclosure will be described with reference to Fig. 13. The musical performance model training process is realized by the above-mentioned musical performance model training device 200, and may be realized by, for example, a processor executing a program or an instruction. Fig. 13 is a flowchart showing the musical performance model training process according to an embodiment of the present disclosure.
[0048] 13, in step S101, the musical performance model training device 200 selects training performance information from the training performance information database 80. Specifically, the musical performance model training device 200 may automatically select the training performance information randomly, sequentially, or by user selection.
[0049] In step S102, the performance model training device 200 converts the performance information into TAB display information.
[0050] In step S103, the performance model training device 200 displays the TAB score on the LCD of the control device 100 or the like.
[0051] In step S104, the performance model training device 200 starts the MIDI player in sync with the tempo of the performance information.
[0052] In step S105, the performance model training device 200 starts the metronome in time with the tempo.
[0053] In step S106, the MIDI player reproduces the performance information.
[0054] In step S107, the metronome reproduces the performance information, which completes the preparations for acquiring the performer's performance, and the performer begins playing.
[0055] In step S108, the performance model training device 200 initializes the string number s to 0. Since the guitar 50 has six strings, the string number s can take a value from 0 to 5.
[0056] In step S109, the performance model training device 200 stores the sound information of the s channel generated from the MIDI player in the sound information memory p.
[0057] In step S110, the performance model training device 200 acquires, from the buffer, a string vibration waveform of string number s that represents the performance by the performer, generates a spectrum feature data frame, and stores it in a ring buffer or the like.
[0058] In step S111, the performance model training device 200 inputs the spectral feature data frame at the reference time, p spectral feature data frames immediately before the reference time, and n spectral feature data frames immediately after the reference time to the performance model to be trained.
[0059] In step S112, the musical performance model training device 200 stores the output result of the musical performance model in the memory o.
[0060] In step S113, the musical performance model training device 200 compares the sound production information (eg, rendition style number and note number) in the memory p with the output result of the musical performance model in the memory o.
[0061] In step S114, the performance model training device 200 judges whether there is a difference between the sound production information in the memory p and the output result of the memory o.
[0062] If there is a significant difference (S114: Yes), the musical performance model training device 200 applies the optimization information for updating the musical performance model from the difference to the musical performance model in step S115, and proceeds to step S116. On the other hand, if there is no significant difference (S114: No), the musical performance model training device 200 proceeds to step S116 without updating the musical performance model.
[0063] In step S116, the performance model training device 200 increments the string number s by 1 to process the next string.
[0064] In step S117, the musical performance model training device 200 determines whether the musical performance model updating process has been completed for all strings, and if the updating process has not been completed for all strings (S117: Yes), the process returns to step S109.
[0065] In step S118, the musical performance model training device 200 judges whether the musical performance information has been processed in its entirety, and if the musical performance information has not been processed in its entirety (S118: No), the process returns to step S106.
[0066] In step S119, the performance model training device 200 stops the metronome and the MIDI player.
[0067] In step S120, the performance model training device 200 determines whether an end operation has been performed by the user or the like. If no end operation has been performed (S120: No), the performance model training device 200 returns to step S101 and selects the next performance information. If an end operation has been performed (S120: Yes), the processing ends. [Performance Information Prediction Device] Next, the performance information prediction device 300 according to an embodiment of the present disclosure will be described with reference to Figures 14 to 16. Figure 14 is a schematic diagram showing the operation of the performance information prediction device 300 according to an embodiment of the present disclosure.
[0068] The performance information prediction device 300 predicts performance information (e.g., MIDI messages, etc.) from the performance of the guitar 50 by the player, using the performance model trained by the performance model training device 200. Specifically, as shown in FIG. 14, when the performance information prediction device 300 acquires string vibration waveform data representing the guitar performance from the guitar 50, it performs a fast Fourier transform on the acquired string vibration waveform data to generate a spectrum data frame of a predetermined window width having an overlapping portion on the time axis. Then, the performance information prediction device 300 identifies a predetermined number of top peaks in terms of frequency in each spectrum data frame, and generates a spectrum feature data frame by extracting the identified peaks. For example, these preprocessing processes may be realized by a DSP.
[0069] In order to predict the performance information at the reference time, the performance information prediction device 300 inputs the spectral feature data frames of a certain time range before and after the reference time, i.e., the spectral feature data frame at the reference time, p spectral feature data frames immediately before the reference time, and n spectral feature data frames immediately after the reference time, into the trained performance model to obtain sound information including the playing style and note number. The performance information prediction device 300 also performs volume detection and pitch detection on the spectrum at the reference time to detect the volume and pitch of the performance. The performance information prediction device 300 generates mute information and pitch change information based on the detected volume and pitch, respectively, and generates velocity information indicating the strength of the plucking based on the volume and adds it to the sound information. In this way, the performance information prediction device 300 generates performance information (e.g., MIDI message, etc.) at each time including sound information, mute information, and / or pitch change information from the spectral feature data frame at each time, and transmits it to an external device (e.g., a playback device, a computer, etc.). For example, these performance information generation processes may be realized by a CPU.
[0070] FIG. 15 is a block diagram showing a functional configuration of a performance information prediction device 300 according to an embodiment of the present disclosure.
[0071] As shown in FIG. 15, the performance information prediction device 300 includes a preprocessing unit 310 and a performance information prediction unit 320.
[0072] The preprocessing unit 310 generates a spectral data frame from string vibration waveform data representing a stringed instrument performance, and converts the spectral data frame into a spectral feature data frame. As with the preprocessing unit 210, when the guitar 50 is played by a performer, the preprocessing unit 310 acquires string vibration waveform data of each string from the guitar 50, performs a fast Fourier transform (FFT) on the string vibration waveform data of each string, and acquires spectral data. Specifically, under the same settings as the preprocessing unit 210, the preprocessing unit 310 may extract string vibration waveform frames of a window width w that overlap with respect to the time axis from the string vibration waveform data, perform an FFT for each sampling, and convert each string vibration waveform frame into a spectral data frame. After conversion into a spectral data frame, the preprocessing unit 310 generates a spectral feature data frame based on a predetermined number of top peaks of each spectral data frame, and stores the spectral feature data frames in a certain time range around the reference time in a ring buffer or the like.
[0073] The performance information prediction unit 320 predicts performance information of a stringed instrument performance from the spectral feature data frame at the reference time and the spectral feature data frames before and after the spectral feature data frame at the reference time using the trained performance model. Specifically, the performance information prediction unit 320 inputs the spectral feature data frame at the reference time, p spectral feature data frames immediately before the reference time, and n spectral feature data frames immediately after the reference time to the trained performance model, and obtains the playing style and note number at the reference time.
[0074] In parallel with this, the performance information prediction unit 320 performs volume detection and pitch detection on the spectral feature data frame at the reference time. For example, for volume detection, the performance information prediction unit 320 may calculate the sum of the frequency levels of a predetermined number of peaks in the spectral feature data frame as shown in FIG. 16(a), and determine the calculated total frequency level as the volume at the reference time. If the trained performance model does not detect an onset at the reference time, or if the detected volume is equal to or less than a predetermined threshold that is recognized as a mute state, the performance information prediction unit 320 determines that there was no onset and outputs mute information as the performance information. Otherwise, the performance information prediction unit 320 determines that there was an onset, sets the detected volume as the velocity value of the onset, and includes the velocity value in the onset information together with the playing style and note number output from the performance model.
[0075] As for pitch detection, the performance information prediction unit 320 determines the minimum frequency level among a predetermined number of peaks in the spectrum feature data frame as the pitch of the plucked string and generates pitch information, as shown in Fig. 16(b). If there is a difference from the most recent pitch information or sound generation information, the performance information prediction unit 320 determines that a pitch change has occurred and outputs pitch change information.
[0076] The pre-processing unit 310 and the performance information prediction unit 320 execute the above-mentioned processes in parallel for all the strings. [Performance information prediction processing] Next, a performance information prediction process according to an embodiment of the present disclosure will be described with reference to Fig. 17. The performance information prediction process is realized by the above-mentioned performance information prediction device 300, and may be realized by, for example, a processor executing a program or an instruction. Fig. 17 is a flowchart showing the performance information prediction process according to an embodiment of the present disclosure.
[0077] 17, in step S201, the performance information prediction device 300 initializes the string number s to 0. Since the guitar 50 is made up of six strings, the string number s takes a value from 0 to 5.
[0078] In step S202, the performance information prediction device 300 generates a spectral feature data frame from the string vibration waveform data, and performs volume detection on the spectral feature data frame at the reference time.
[0079] In step S203, the performance information prediction device 300 determines whether the detected volume I is less than a predetermined threshold. If the volume I is equal to or greater than the predetermined threshold (S203: No), the performance information prediction device 300 determines that sound is being generated, and proceeds to step S206.
[0080] On the other hand, if the volume I is less than the predetermined threshold (S203: Yes), the performance information prediction device 300 determines in step S204 whether the note is being sounded at the reference time. For example, the determination may be made based on whether sounding information has been output from the trained performance model in the previous time. If the note is being sounded (S204: Yes), the performance information prediction device 300 generates mute information corresponding to the note number output from the performance model in step S205. If the note is not being sounded (S204: No), the performance information prediction device 300 proceeds to step S206.
[0081] In step S206, the performance information prediction device 300 extracts, from the buffer, the spectral feature data frame at the reference time, p spectral feature data frames immediately before the reference time, and n spectral feature data frames immediately after the reference time.
[0082] In step S207, the performance information prediction device 300 inputs the extracted spectrum feature data frame into the performance model.
[0083] In step S208, the performance information prediction device 300 judges whether or not sound information including a playing style and a note number has been output from the performance model. If sound information has been output (S208: Yes), the performance information prediction device 300 assigns the output playing style and note number to variables v and k, respectively, in step S209. On the other hand, if sound information has not been output (S208: No), the performance information prediction device 300 proceeds to step S215.
[0084] In step S210, the performance information prediction device 300 determines whether or not a sound has been produced. If a sound has been produced (S210: Yes), the performance information prediction device 300 generates mute information for the note number K0 of the previous sound production event in step S211. On the other hand, if a sound has not been produced (S210: No), the performance information prediction device 300 proceeds to step S212.
[0085] In step S212, the musical performance information prediction device 300 generates sound production information including a velocity converted from the musical performance style number v, the note number k, and the volume I.
[0086] In step S213, the performance information prediction device 300 assigns k to the note number K0 of the previous sound generation event.
[0087] In step S214, the performance information prediction device 300 identifies a pitch corresponding to the previously occurring pitch P0=k.
[0088] In step S215, the performance information prediction device 300 performs pitch detection on the spectral feature data frame at the reference time, and stores the detected pitch in p.
[0089] In step S216, the performance information prediction device 300 judges whether p=P0. If p=P0 (S216: Yes), the performance information prediction device 300 proceeds to step S218, and if p=P0 is not satisfied (S216: No), in step S217, the performance information prediction device 300 generates pitch bend information based on the difference from p.
[0090] In step S218, the performance information prediction device 300 increments the string number s by 1 to execute the above-mentioned process for the next string.
[0091] In step S219, the performance information prediction device 300 determines whether all strings have been processed. If all strings have been processed (S219: No), the performance information prediction process ends. If not (S219: Yes), the performance information prediction device 300 repeats the above-described process for the next string.
[0092] In the above embodiment, a performance information prediction system is described in which a performance model that predicts performance information from string vibration waveform data of a stringed instrument such as the guitar 50 is trained, and the performance information is predicted using the trained performance model, but the present disclosure is not limited to this and may be applied to a wind instrument. That is, the present disclosure may be applied to a performance information prediction system in which a performance model that predicts performance information from air vibration waveform data of a wind instrument is trained, and the performance information is predicted using the trained performance model.
[0093] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the specific embodiments described above, and various modifications and variations are possible within the scope of the gist of the present invention as defined in the claims.
[0094] The invention as originally claimed in the present application is set forth below. [Note] In one aspect of the present disclosure, a pre-processing unit that generates a spectral data frame from string vibration waveform data representing a stringed instrument performance, and obtains a spectral characterization data frame based on the spectral data frame; a performance information prediction unit that predicts performance information of the stringed instrument performance from a spectral feature data frame at a reference time and spectral feature data frames before and after the spectral feature data frame at the reference time by using a trained performance model; The present invention provides a performance information prediction device having the above structure.
[0095] In one embodiment, the performance information may consist of sound generation information, sound muting information and pitch modification information.
[0096] In one embodiment, the sound information may include a playing style and a note number.
[0097] In one embodiment, the trained performance model may output the playing style and the note number.
[0098] In one embodiment, the trained performance model may obtain a spectral feature data frame at the reference time, a first number of spectral feature data frames before the spectral feature data frame at the reference time, and a second number of spectral feature data frames after the spectral feature data frame at the reference time, and output the playing style and the note number.
[0099] In one embodiment, the trained performance model may be implemented as a neural network.
[0100] In one embodiment, the spectral characterization data frame may consist of a predetermined number of the most significant peaks contained in the spectral data frame.
[0101] In one embodiment, the performance information may be described according to the MIDI protocol.
[0102] In another aspect of the present disclosure, a pre-processing unit that generates a spectral data frame from string vibration waveform data representing a stringed instrument performance performed in accordance with training performance information, and obtains a spectral characterization data frame based on the spectral data frame; a performance model training unit that uses the training performance information to train a performance model that predicts performance information of the stringed instrument performance from a spectral feature data frame at a reference time and spectral feature data frames before and after the spectral feature data frame at the reference time; A performance model training device having the above structure is provided.
[0103] In another aspect of the present disclosure, Electronic string instruments, The performance information prediction device described above; The above-mentioned performance model training device; A performance information generating system having the above structure is provided.
[0104] In another aspect of the present disclosure, generating, by one or more processors, a spectral data frame from string vibration waveform data representative of a stringed instrument performance, and obtaining a spectral characterization data frame based on the spectral data frame; predicting performance information of the stringed instrument performance from a spectral feature data frame at a reference time and spectral feature data frames before and after the spectral feature data frame at the reference time using a trained performance model; A performance information prediction method is provided, which has the following steps:
[0105] In another aspect of the present disclosure, generating, by one or more processors, a spectral data frame from string vibration waveform data representative of a stringed instrument performance performed according to training performance information, and obtaining a spectral characterization data frame based on the spectral data frame; training a performance model using the training performance information, the one or more processors, to predict performance information of the stringed instrument performance from a spectral feature data frame at a reference time and spectral feature data frames before and after the spectral feature data frame at the reference time; A performance model training method is provided, which has the following: [Explanation of symbols]
[0106] 10 Guitar Controller 50 Guitar 100 Control device 200 Performance model training device 210 Pretreatment section 220 Performance Model Training Department 300 Performance Information Prediction Device 310 Pretreatment section 320 Performance Information Prediction Unit
Claims
1. a pre-processing unit that generates a spectral data frame from string vibration waveform data representing a stringed instrument performance, and obtains a spectral characterization data frame based on the spectral data frame, the spectral characterization data frame being configured from the frequencies of a predetermined number of top peaks included in the spectral data frame; a performance information prediction unit that predicts performance information of the stringed instrument performance from the spectral feature data frame at a reference time and the spectral feature data frames at times before and after the reference time by using a trained performance model; having The trained performance model obtains the spectral feature data frame at the reference time, a first number of spectral feature data frames before the reference time, and a second number of spectral feature data frames after the reference time, and outputs a playing style and note number of a stringed instrument.
2. 2. The performance information prediction device according to claim 1, wherein the performance information comprises sound generation information, sound mute information, and pitch change information.
3. 3. The performance information prediction device according to claim 2, wherein the sound generation information includes the playing style and the note number.
4. 4. The apparatus according to claim 1, wherein the trained performance model is realized by a neural network.
5. 5. The performance information prediction device according to claim 1, wherein the performance information is described in accordance with a MIDI protocol.
6. a pre-processing unit that generates a spectral data frame from string vibration waveform data representing a stringed instrument performance performed in accordance with training performance information, and obtains a spectral characterization data frame based on the spectral data frame, the spectral characterization data frame being configured from frequencies of a predetermined number of upper peaks included in the spectral data frame; a performance model training unit that uses the training performance information to train a performance model that predicts performance information of the stringed instrument performance from the spectral feature data frame at a reference time and the spectral feature data frames at times before and after the reference time; A performance model training device having the above structure.
7. Electronic string instruments, The performance information prediction device according to any one of claims 1 to 5, A musical performance model training device according to claim 6, A performance information generating system having the above configuration.
8. one or more processors generating a spectral data frame from string vibration waveform data representing a stringed instrument performance, and obtaining a spectral characterization data frame based on the spectral data frame, the spectral characterization data frame being composed of frequencies of a predetermined number of top peaks included in the spectral data frame; The one or more processors predict performance information of the stringed instrument performance from the spectral feature data frame at a reference time and the spectral feature data frames at times around the reference time using a trained performance model; having The trained performance model obtains the spectral feature data frame at the reference time, a first number of spectral feature data frames before the reference time, and a second number of spectral feature data frames after the reference time, and outputs a playing style and note number of a stringed instrument.
9. one or more processors generating a spectral data frame from string vibration waveform data representing a stringed instrument performance performed according to training performance information, and obtaining a spectral characterization data frame based on the spectral data frame, the spectral characterization data frame being composed of frequencies of a predetermined number of top peaks included in the spectral data frame; training a performance model using the training performance information, the one or more processors, to predict performance information of the stringed instrument performance from the spectral feature data frame at a reference time and the spectral feature data frames at times around the reference time; A performance model training method having the above structure.
Citation Information
Patent Citations
Musical sound recognition device
JP1993188932A
Detection method of musical performance position and detection method of pitch
JP1997006339A
Chord vibration pitch detecting device and chord vibration pitch detecting method
JP2000105590A
Sound waveform extraction device, and program
JP2010217408A
System and method for capturing and interpreting audio
JP2017532614A
