Performance information prediction device, performance model training device, performance information generation system, performance information prediction method, and performance model training method

By generating waveform characterization data and using a trained performance model, the guitar controller accurately predicts performance information, addressing latency and expressiveness issues in conventional guitar-based MIDI controllers.

JP7585682B2Active Publication Date: 2024-11-19CASIO COMPUTER CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2020158866
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-09-23
Publication Date
2024-11-19
Estimated Expiration
2040-09-23

AI Technical Summary

Technical Problem

Conventional guitar-based MIDI controllers suffer from latency issues in bass pitch detection, misrecognition of pitches, and lack of expressiveness in capturing playing styles, leading to inaccurate performance information conversion.

Method used

A preprocessing unit generates waveform characterization data from string vibration waveforms, using a trained performance model to predict performance information, including playing style and note number, through a neural network or recurrent neural network, reducing latency and improving accuracy.

Benefits of technology

The solution enables high-precision conversion of electronic stringed instrument performance into performance information, reducing latency and accurately detecting playing styles and pitch changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007585682000001
    Figure 0007585682000001
  • Figure 0007585682000002
    Figure 0007585682000002
  • Figure 0007585682000003
    Figure 0007585682000003
Patent Text Reader

Abstract

To provide a technique that can precisely convert a musical performance of an electronic string musical instrument to musical performance information.SOLUTION: A musical performance information prediction device according to an embodiment of the present disclosure includes: a pre-treatment unit for generating waveform featuring data including a zero-cross point, the smallest point, and the largest point from string vibration waveform data showing a string musical instrument performance; and a musical performance information prediction unit for predicting musical performance information of the string musical instrument performance from the waveform featuring data by using a trained playing model.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a performance information prediction device, a performance model training device, a performance information generating system, a performance information prediction method, and a performance model training method. [Background technology]

[0002] There is an electronic musical instrument called a guitar controller (or guitar synthesizer) that converts the string vibration waveform of an instrument such as a guitar into an electric signal using a magnetic or piezo pickup, and then analyzes the pitch and volume to convert it into digital performance data such as MIDI (Musical Instrument Digital Interface) messages. Unlike dedicated guitar controllers that are only used to play sound sources, this type of controller has the great advantage of being able to play MIDI by equipping each string with an independent pickup to obtain performance information while retaining the shape and functions of a normal guitar, and is the most common form. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 9-6339 [Patent Document 2] JP 2000-105590 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, one of the major problems that has not been solved for many years with such instruments is the latency of detecting the pitch of low notes when the strings are plucked. Conventional guitar / bass MIDI controllers detect the pitch of low notes using observation methods such as autocorrelation and zero-crossing judgment, which requires multiple periods of judgment data. For example, the frequency of the lowest note E2 on a six-string guitar tuned to A4=440Hz is 82.4Hz. Of course, it is possible to generate lower notes using the tremolo arm, lower the pitch of the low strings using alternative tuning, or generate even lower frequencies with a multi-string guitar with additional low strings. However, even with this frequency of 82.4Hz, one waveform period is about 12msec, and conventional pitch detection methods have to wait 24-36msec to determine the pitch after comparing two to three waveform periods and determining that the waveforms are almost the same pattern. Furthermore, when picking hard, or when playing with a special waveform that has no periodicity, such as with slap or pluck techniques, the pitch cannot be detected until the vibration stabilizes, and it can take tens of milliseconds or more before sound is produced. This is a latency that cannot be ignored when playing.

[0005] If the judgment is made at a short period in order to reduce latency, there is a dilemma that the possibility of false pitch detection increases, so in the end it is a trade-off between speed and accuracy.

[0006] As described above, compared to keyboard controllers, guitar controllers can have poor response to playing in certain ranges and can misrecognize pitch, so the reality is that you have to be much more careful than when playing a real guitar, and the content of what you can do is limited, but there is still a risk of making a mistake.

[0007] Another problem is that while conventional methods detect pitch and level, they do not detect playing technique, or if they do, they only detect changes in pitch and volume.

[0008] Specifically, there are many ways to produce sound, such as normal picking, plucking the strings with the fingers, thumping with the thumb, pulling the strings and then releasing them, open harmonics, picking harmonics, hammering on, pulling off, tapping, and more. However, with conventional devices, the same performance information was generated regardless of which of these techniques was used, resulting in a loss of expressiveness.

[0009] In view of the above problems, an object of the present disclosure is to provide a technology for converting the performance of an electronic stringed instrument into performance information with high accuracy. [Means for solving the problem]

[0010] In order to solve the above problems, one aspect of the present disclosure provides a pre-processing unit that generates waveform characterization data including zero crossing points, minimum points, and maximum points from string vibration waveform data representing a stringed instrument performance; Outputting a stringed instrument playing style and note number from the waveform characterization data and a performance information prediction unit that predicts performance information of the stringed instrument performance from the waveform feature data by using a trained performance model. Effect of the Invention

[0011] According to the present disclosure, the performance of an electronic stringed instrument can be converted into performance information with high accuracy. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram illustrating a guitar controller according to one embodiment of the present disclosure. [Diagram 2] A diagram showing the configuration of performance information according to one embodiment of the present disclosure. [Diagram 3] FIG. 1 illustrates a tablature according to an embodiment of the present disclosure. [Figure 4] FIG. 1 illustrates an external view of a guitar controller according to an embodiment of the present disclosure. [Diagram 5] FIG. 2 is a block diagram showing a hardware configuration of a guitar according to an embodiment of the present disclosure. [Figure 6]FIG. 2 is a block diagram showing a hardware configuration of a control device according to an embodiment of the present disclosure. [Figure 7] 1 is a schematic diagram illustrating an operation of a musical performance model training device according to an embodiment of the present disclosure. [Figure 8] FIG. 1 illustrates a string vibration waveform according to one embodiment of the present disclosure. [Figure 9] FIG. 1 illustrates a string vibration waveform according to one embodiment of the present disclosure. [Figure 10] FIG. 1 illustrates a string vibration waveform according to one embodiment of the present disclosure. [Figure 11] 1 is a block diagram showing a functional configuration of a musical performance model training device according to an embodiment of the present disclosure. [Figure 12] FIG. 2 is a schematic diagram illustrating waveform characterization according to an embodiment of the present disclosure. [Figure 13] FIG. 2 is a schematic diagram illustrating filtering according to an embodiment of the present disclosure. [Figure 14] FIG. 2 is a schematic diagram illustrating waveform characterization data according to an embodiment of the present disclosure. [Figure 15] A diagram showing the architecture of a performance model according to one embodiment of the present disclosure. [Figure 16] A diagram showing the architecture of a performance model according to another embodiment of the present disclosure. [Figure 17] 13 is a flowchart illustrating a performance model training process according to an embodiment of the present disclosure. [Figure 18] 1 is a schematic diagram illustrating an operation of a performance information prediction device according to an embodiment of the present disclosure. [Figure 19] 1 is a block diagram showing a functional configuration of a performance information prediction device according to an embodiment of the present disclosure. [Figure 20] FIG. 2 is a schematic diagram illustrating volume detection and pitch detection according to one embodiment of the present disclosure. [Figure 21] 11 is a flowchart illustrating a performance information prediction process according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] In the following embodiment, a guitar controller that generates performance information (e.g., MIDI messages) from a string vibration waveform generated by playing a guitar is disclosed. Note that the present disclosure is not limited to a guitar controller, and may be applied to any other performance information generating device that generates performance information from the performance of a stringed instrument having a string vibration waveform extraction function. [Summary of this disclosure] 1, a guitar controller 10 according to an embodiment of the present disclosure includes a guitar 50 and a control device 100. The guitar controller 10 generates performance information from a string vibration waveform generated by playing the guitar 50, using a performance model realized as a machine learning model such as a neural network.

[0014] As shown in FIG. 2, the performance information according to an embodiment of the present disclosure indicates the performance type of sound generation information, sound muting information, and pitch change information.

[0015] The sound generation information indicates the note number and playing style determined by the performance model as a classification model, and the strength of the string plucking determined by envelope detection. The playing styles are classified into seven types, for example: 0) picking with a pick, 1) finger picking, 2) hammer-on (tapping), 3) pulling off, 4) mute picking, 5) open harmonics, and 6) picking harmonics. When sound generation is expressed by MIDI messages, the playing style may be represented by Control Change: 0xBn, 0x46, vv, and the note number and string plucking strength may be represented by Note On: 0x9n, kk, vv.

[0016] Moreover, the mute information is detected by envelope detection, and indicates 0) sound stop and 1) replacement. When mute is expressed by a MIDI message, it may be expressed by Control Change: 0xBn, 0x46, vv and Note Off: 0x8n, kk, vv.

[0017] The pitch change information is detected by zero cross counting, and indicates, for example, half-tone bend up, half-tone bend down, whole-tone bend up, whole-tone bend down, one-and-a-half-tone bend up, one-and-a-half-tone bend down, two-tone bend up, two-tone bend down, and slide. When expressing pitch change by MIDI message, it may be expressed by Pitch Bend: 0xEn,ll,mm.

[0018] In the embodiment shown in FIG. 1, the guitar controller 10 has two operating modes: a training mode for training a performance model, and a performance mode for predicting performance information using the trained performance model. The control device 100 has a performance model training device 200 used in the training mode, and a performance information prediction device 300 used in the performance mode.

[0019] First, in the training mode, the guitar controller 10 acquires training data from the training performance information database 80. The training data is composed of, for example, a pair of music score data (for example, TAB scores, etc.) and a MIDI file corresponding to the music score data. The TAB scores may be written according to a well-known notation, for example, as shown in FIG. 3. When the user plays the guitar 50 according to the music score of the acquired training music score data, the performance model training device 200 inputs string vibration information generated by the guitar 50 based on the user's performance to the performance model to be trained, compares the MIDI message as performance information output from the performance model with the training MIDI file, and trains the performance model so that the error between them is reduced. In the present disclosure, the string vibration information indicating the relationship between time and amplitude is high-pass filtered and low-pass filtered, and performance information is acquired from the performance model using waveform characterization data characterized based on zero crossing points, minimum points, and maximum points in the filtered string vibration waveform. When training is completed, the performance model training device 200 provides the trained performance model to the performance information prediction device 300.

[0020] Next, in the performance mode, when the user plays the guitar 50, the performance information prediction device 300 inputs string vibration information generated by the guitar 50 based on the user's performance into the trained performance model, and acquires performance information such as MIDI messages. The acquired performance information is transmitted to, for example, an external playback device or computer, and the user can play back the performance by the user via the playback device or use the performance information on the computer.

[0021] This reduces the delay in pitch detection when converting the performance of an electronic stringed instrument into performance information, and makes it possible to determine the playing style with high accuracy.

[0022] It should be noted that the guitar controller 10 according to the embodiment described below includes the performance model training device 200, but the present disclosure is not limited thereto. For example, the performance model may be trained by an external computer or server, and the trained performance model and / or update information of the performance model may be provided to the performance information prediction device 300 from the external computer or server. [Hardware configuration] Next, the physical configuration of the guitar controller 10 will be described with reference to Fig. 4. Fig. 4 is a diagram showing the external appearance of the guitar controller 10 according to one embodiment of the present disclosure.

[0023] As shown in FIG. 4, the guitar controller 10 is a separate type performance information generating system that is composed of a guitar 50 and a control device 100 that are interconnected.

[0024] The guitar 50 is a normal electric guitar equipped with a hexa-divided pickup for picking up the independent vibrations of each of the six strings, a MIDI volume for controlling the volume of the performance information, and an up / down switch for switching the patch memory number up or down for the control device 100. This information and the output of the normal pickup are sent to the control device 100 via a multi-cable. Power is also supplied from the control device 100 via the multi-cable. The hexa-divided pickup of this embodiment is a magnetic pickup, the same as a normal pickup.

[0025] On the other hand, the control device 100 receives input of guitar string vibrations and generates performance information in MIDI format. The destination of the performance information may be, without limitation, a sound source unit, a computer, etc. As shown in FIG. 1, the control device 100 has a foot switch for switching the bank number and number of the patch memory in which various settings are stored, a CONTROL switch and a foot pedal that can assign and transmit any performance message. The number of the currently selected patch memory is displayed on the BANK / NUM screen. An LCD is used as the main display device, but a touch panel is also attached on the screen. A rotary encoder for inputting data is also provided on the panel. As terminals, the device is provided with a GUITAR INPUT input terminal for a multi-cable from the guitar 50, a GUITAR OUT audio output terminal for a normal pickup, a MIDI OUT output terminal for a MIDI performance signal, a USB to HOST connection terminal for a host computer, and an AC power input terminal AC POWER.

[0026] Next, a hardware configuration of a guitar 50 according to an embodiment of the present disclosure will be described with reference to Fig. 5. Fig. 5 is a block diagram showing a hardware configuration of a guitar 50 according to an embodiment of the present disclosure.

[0027] 5, from the guitar 50, the signal passed through the buffer amplifier of the hexa-divided pickup, the MIDI volume control, the up / down switch of the patch memory, and the signal of the normal pickup are sent to the control device 100 via a multi-cable. The three normal pickups are selected by the pickup selector, and the signal passed through the buffer amplifier via the tone control circuit and the volume control circuit is sent to the control device 100.

[0028] Next, a hardware configuration of the control device 100 according to an embodiment of the present disclosure will be described with reference to Fig. 6. Fig. 6 is a block diagram showing a hardware configuration of the control device according to an embodiment of the present disclosure.

[0029] As shown in FIG. 6, the control device 100 is composed of a CPU (Central Processing Unit) and a DSP (Digital Signal Processor). The CPU manages the overall functions and processing of the control device 100, and the DSP executes waveform analysis processing that requires high-speed processing. The CPU bus is connected to the RAM used by the CPU, the Flash ROM, the LCD controller that controls the LCD, the I / O interface connected to various I / O devices, the DSP, the USB interface, and the MIDI interface. Furthermore, the I / O interface is connected to a foot switch, a rotary encoder, an LCD touch panel, the MIDI volume of the guitar 50, an A / D converter for detecting the position of the pedal of the control device 100, and an LED for displaying the patch memory number. Although only one A / D converter is shown in the figure, the input source is switched in a time-division manner by a multiplexer to read the value. The DSP, to which a dedicated RAM and Flash ROM are connected, is connected to an independent A / D converter for quickly converting the output of the six strings of the hexa-divided pickup into a digital signal, enabling high-speed analysis processing.

[0030] However, the guitar 50 and the control device 100 are not limited to the above-mentioned hardware configuration, and may be realized by any other appropriate hardware configuration. [Performance model training device] Next, the musical performance model training device 200 according to an embodiment of the present disclosure will be described with reference to Figures 7 to 16. Figure 7 is a schematic diagram showing the operation of the musical performance model training device 200 according to an embodiment of the present disclosure.

[0031] As shown in FIG. 7, the performance model training device 200 trains the performance model using training data stored in the training performance information database 80 that stores the training data. Specifically, the training data is composed of a pair of training score data and training performance information corresponding to the score data, and a score (e.g., TAB score, etc.) displayed based on the training score data is displayed to the performer, and the performer plays the guitar 50 under tempo control by a metronome. String vibration waveform data representing the performance is provided to the performance model training device 200, which converts the acquired string vibration waveform data into waveform feature data, which will be described in detail below, and inputs the waveform feature data to the performance model to be trained. Then, the performance model training device 200 compares the output from the performance model with the sound information (e.g., note number and playing style, etc.) of the training performance information, and updates the parameters of the performance model according to the difference. The performance model training device 200 repeats the above-mentioned process until a predetermined end condition is satisfied, and optimizes the performance model so that the output from the performance model approaches the sound information of the training performance information.

[0032] In this disclosure, attention is focused on the leading waveform when the string is plucked, and the playing style and note number are determined based on the difference in the shape characteristics of the leading waveform. For example, Fig. 8(a)-(c) show leading waveform patterns according to pitch, and show the waveform change due to the fret position, i.e., the difference in pitch. As can be seen from the figures, the time interval between the minimum point and the maximum point changes. Fig. 9(a)-(c) show leading waveform patterns according to the strength of the string plucking, and show the waveform change due to the difference in the strength of the string plucking. As can be seen from the figures, the amplitude difference between the minimum point and the maximum point changes. Fig. 10(a)-(g) show leading waveform patterns according to the playing style, and the positions and amplitudes of the zero crossing point, the minimum point, and the maximum point change. Thus, it is considered that there is a correlation between the zero crossing point, the minimum point, and the maximum point of the leading waveform when the string is plucked, and the playing style and note number.

[0033] FIG. 11 is a block diagram showing a functional configuration of a musical performance model training device 200 according to an embodiment of the present disclosure.

[0034] As shown in FIG. 11, the musical performance model training device 200 includes a preprocessing unit 210 and a musical performance model training unit 220 .

[0035] The pre-processing unit 210 generates waveform characterization data consisting of zero crossing points, minimum points, and maximum points from string vibration waveform data representing a stringed instrument performance.

[0036] Specifically, when the guitar 50 is played by a player, the guitar 50 acquires string vibration waveform data indicating time and the amplitude of each string as shown in Fig. 12(a) and transmits it to the performance model training device 200. That is, since the guitar 50 has six strings, six types of string vibration waveform data are generated. For example, the string vibration waveform data can be acquired at a sampling rate of 20 KHz (at 50 μsec intervals) and a data word length of 16 bits, which is sufficient for analyzing string vibration.

[0037] The pre-processing unit 210 performs low-pass filtering (LPH) and high-pass filtering (HPF) on the string vibration waveform data of each string, as shown in Fig. 12(b). The low-pass filter cuts out unnecessary harmonic components and is useful for extracting unnecessary minimum and maximum points, as shown in Fig. 13(a). The high-pass filter cuts out unnecessary DC components and is useful for extracting zero-crossing points, as shown in Fig. 13(b).

[0038] Then, the pre-processing unit 210 extracts zero crossing points, minimum points, and maximum points from the filtered string vibration waveform, as shown in Fig. 12(c). Here, as shown in Fig. 14, the zero crossing point Zn is a point where the amplitude becomes zero, and the coordinate of the zero crossing point is represented by Zn(x,0). The coordinate of the minimum point is represented by Ln(x,y), and the coordinate of the maximum point is represented by Mn(x,y). Here, the X coordinate represents the number of samples from the beginning, and the Y coordinate is held in 16 bits, which is the resolution of the A / D converter. The extracted zero crossing points, minimum points, and maximum points have a higher priority as the absolute value of their amplitude values ​​increases.

[0039] The pre-processing unit 210 generates waveform characterization data consisting of the zero crossing points, minimum points, and maximum points extracted in this manner, and provides the data to the performance model training unit 220 .

[0040] The performance model training unit 220 predicts performance information of a stringed instrument from the waveform feature data using a trained performance model. Here, the performance model to be trained outputs a playing style and a note number from the coordinates of the zero crossing point, the minimum point, and the maximum point. Specifically, the performance model training unit 220 inputs the coordinates Zn of the zero crossing point, the minimum point Ln, and the maximum point Mn to the performance model to be trained, and obtains the playing style Var and the note number Note.

[0041] In one embodiment, the performance model may be realized by a neural network. For example, the performance model may be a neural network having a network architecture as shown in Fig. 15. In this case, the performance model training unit 220 inputs the coordinates Zn of the zero crossing point, the minimum point Ln, and the maximum point Mn to the input layer of the neural network, and obtains the rendition style number Var and the note number Note from the output layer through calculations in the intermediate layer.

[0042] In another embodiment, the performance model may be realized by a recurrent neural network as shown in Fig. 16. In this case, the performance model training unit 220 inputs the waveform feature data at the reference time t, the b pieces of waveform feature data immediately before the reference time, and the f pieces of waveform feature data immediately after the reference time to the input layer X t-b ,···X t-1 ,X t ,X t+1 ,···,X t+f and obtain the rendition style number Var and the note number Note from the output layer through calculations in the intermediate layer. A recurrent neural network is suitable for processing time-series data, and is considered to be able to predict the rendition style number Var and the note number Note with high accuracy.

[0043] The performance model to be trained may be a pre-trained machine learning model, and the performance model training unit 220 may fine-tune the pre-trained performance model by the above-mentioned training process. This makes it possible to build a highly accurate performance model with less training data than training a performance model from an initial machine learning model.

[0044] When the performance model training unit 220 acquires the playing style and note number from the performance model, the performance model training unit 220 compares the acquired playing style and note number with the playing style and note number of the training performance information, and updates the parameters of the performance model so that they match. For example, when the performance model is realized by a neural network, the performance model training unit 220 may update the parameters of the neural network according to the comparison result according to the well-known backpropagation method.

[0045] The performance model training unit 220 repeats the above-mentioned process to train the performance model until a predetermined termination condition is satisfied, and when the predetermined termination condition is satisfied, passes the performance model at that time point as a trained performance model to the performance information prediction device 300. Here, the predetermined termination condition may be that all prepared training data has been processed, or the like. [Performance model training process] Next, a musical performance model training process according to an embodiment of the present disclosure will be described with reference to Fig. 17. The musical performance model training process is realized by the above-mentioned musical performance model training device 200, and may be realized by, for example, a processor executing a program or an instruction. Fig. 17 is a flowchart showing the musical performance model training process according to an embodiment of the present disclosure.

[0046] 17, in step S101, the musical performance model training device 200 selects training performance information from the training performance information database 80. Specifically, the musical performance model training device 200 may automatically select the training performance information randomly, sequentially, or by user selection.

[0047] In step S102, the performance model training device 200 converts the performance information into TAB display information.

[0048] In step S103, the performance model training device 200 displays the TAB score on the LCD of the control device 100 or the like.

[0049] In step S104, the performance model training device 200 starts the MIDI player in sync with the tempo of the performance information.

[0050] In step S105, the performance model training device 200 starts the metronome in time with the tempo.

[0051] In step S106, the MIDI player reproduces the performance information.

[0052] In step S107, the metronome reproduces the performance information, which completes the preparations for acquiring the performer's performance, and the performer begins playing.

[0053] In step S108, the performance model training device 200 initializes the string number s to 0. Since the guitar 50 has six strings, the string number s can take a value from 0 to 5.

[0054] In step S109, the performance model training device 200 stores the sound information of the s channel generated from the MIDI player in the sound information memory p.

[0055] In step S110, the performance model training device 200 acquires, from the buffer, the string vibration waveform of string number s that represents the performance by the performer, generates waveform characterization data, and stores it in a ring buffer or the like.

[0056] In step S111, the musical performance model training device 200 inputs the waveform feature data at the reference time, the p pieces of waveform feature data immediately before the reference time, and the n pieces of waveform feature data immediately after the reference time to the musical performance model to be trained.

[0057] In step S112, the musical performance model training device 200 stores the output result of the musical performance model in the memory o.

[0058] In step S113, the musical performance model training device 200 compares the sound production information (eg, rendition style number and note number) in the memory p with the output result of the musical performance model in the memory o.

[0059] In step S114, the performance model training device 200 judges whether there is a difference between the sound production information in the memory p and the output result of the memory o.

[0060] If there is a significant difference (S114: Yes), the musical performance model training device 200 applies the optimization information for updating the musical performance model from the difference to the musical performance model in step S115, and proceeds to step S116. On the other hand, if there is no significant difference (S114: No), the musical performance model training device 200 proceeds to step S116 without updating the musical performance model.

[0061] In step S116, the performance model training device 200 increments the string number s by 1 to process the next string.

[0062] In step S117, the musical performance model training device 200 determines whether the musical performance model updating process has been completed for all strings, and if the updating process has not been completed for all strings (S117: Yes), the process returns to step S109.

[0063] In step S118, the musical performance model training device 200 judges whether the musical performance information has been processed in its entirety, and if the musical performance information has not been processed in its entirety (S118: No), the process returns to step S106.

[0064] In step S119, the performance model training device 200 stops the metronome and the MIDI player.

[0065] In step S120, the performance model training device 200 determines whether an end operation has been performed by the user or the like. If no end operation has been performed (S120: No), the performance model training device 200 returns to step S101 and selects the next performance information. If an end operation has been performed (S120: Yes), the processing ends. [Performance Information Prediction Device] Next, the performance information prediction device 300 according to an embodiment of the present disclosure will be described with reference to Figures 18 to 20. Figure 18 is a schematic diagram showing the operation of the performance information prediction device 300 according to an embodiment of the present disclosure.

[0066] The performance information prediction device 300 predicts performance information (e.g., MIDI messages, etc.) from a performer's performance of the guitar 50, using the performance model trained by the performance model training device 200. Specifically, as shown in Fig. 18, when the performance information prediction device 300 acquires string vibration waveform data representing a guitar performance from the guitar 50, it performs high-pass filtering and low-pass filtering on the acquired string vibration waveform data, and generates waveform characterization data extracted from zero-crossing points, minimum points, and maximum points. For example, these pre-processing steps may be realized by a DSP.

[0067] In order to predict the performance information at the reference time, the performance information prediction device 300 inputs waveform feature data, for example, the waveform feature data at the reference time, p pieces of waveform feature data immediately before the reference time, and n pieces of waveform feature data immediately after the reference time, into the trained performance model to obtain sound information including the playing style and note number. The performance information prediction device 300 also performs volume detection and pitch detection on the waveform feature data at the reference time to detect the volume and pitch of the performance. The performance information prediction device 300 generates silencing information and pitch change information based on the detected volume and pitch, respectively, and generates velocity information indicating the strength of the plucking based on the volume and adds it to the sound information. In this way, the performance information prediction device 300 generates performance information (e.g., MIDI message, etc.) at each time including sound information, silencing information, and / or pitch change information from the waveform feature data at each time, and transmits it to an external device (e.g., a playback device, a computer, etc.). For example, these performance information generation processes may be realized by a CPU.

[0068] FIG. 19 is a block diagram showing a functional configuration of a performance information prediction device 300 according to an embodiment of the present disclosure.

[0069] As shown in FIG. 19, the performance information prediction device 300 includes a preprocessing unit 310 and a performance information prediction unit 320.

[0070] The pre-processing unit 310 converts string vibration waveform data representing a stringed instrument performance into waveform characterization data. Like the pre-processing unit 210, when the guitar 50 is played by a player, the pre-processing unit 310 acquires string vibration waveform data of each string from the guitar 50, performs high-pass filtering and low-pass filtering on the string vibration waveform data of each string, and generates waveform characterization data consisting of zero-crossing points, minimum points, and maximum points.

[0071] The performance information prediction unit 320 predicts performance information of a stringed instrument performance from the waveform feature data at a reference time using a trained performance model. Specifically, the performance information prediction unit 320 inputs the waveform feature data at a reference time into the trained performance model and obtains the playing style and note number at the reference time.

[0072] In parallel with this, the performance information prediction unit 320 performs volume detection and pitch detection on the waveform feature data at the reference time. For example, for volume detection, the performance information prediction unit 320 may convert the waveform feature data frame into an absolute value, perform low-pass filtering, and determine the amplitude level of the filtered waveform as the volume at the reference time, as shown in FIG. 20(a). If the trained performance model does not detect an onset at the reference time, or if the detected volume is equal to or less than a predetermined threshold value that is recognized as a mute state, the performance information prediction unit 320 determines that there was no onset and outputs mute information as the performance information. Otherwise, the performance information prediction unit 320 determines that there was an onset, sets the detected volume as the velocity value of the onset, and includes the velocity value in the onset information together with the playing style and note number output from the performance model.

[0073] Regarding pitch detection, the performance information prediction unit 320 counts the zero crossing points of the waveform feature data to generate pitch information, as shown in Fig. 20(b). If there is a difference between the latest pitch information or sound generation information, the performance information prediction unit 320 determines that a pitch change has occurred and outputs pitch change information.

[0074] The pre-processing unit 310 and the performance information prediction unit 320 execute the above-mentioned processes in parallel for all the strings. [Performance information prediction processing] Next, a performance information prediction process according to an embodiment of the present disclosure will be described with reference to Fig. 21. The performance information prediction process is realized by the above-mentioned performance information prediction device 300, and may be realized by, for example, a processor executing a program or an instruction. Fig. 21 is a flowchart showing the performance information prediction process according to an embodiment of the present disclosure.

[0075] 21, in step S201, the performance information prediction device 300 initializes the string number s to 0. Since the guitar 50 is made up of six strings, the string number s takes a value from 0 to 5.

[0076] In step S202, the performance information prediction device 300 generates waveform feature data from the string vibration waveform data, and performs volume detection on the waveform feature data at the reference time.

[0077] In step S203, the performance information prediction device 300 determines whether the detected volume I is less than a predetermined threshold. If the volume I is equal to or greater than the predetermined threshold (S203: No), the performance information prediction device 300 determines that sound is being generated, and proceeds to step S206.

[0078] On the other hand, if the volume I is less than the predetermined threshold (S203: Yes), the performance information prediction device 300 determines in step S204 whether the note is being sounded at the reference time. For example, the determination may be made based on whether sounding information has been output from the trained performance model. If the note is being sounded (S204: Yes), the performance information prediction device 300 generates mute information corresponding to the note number output from the performance model in step S205. If the note is not being sounded (S204: No), the performance information prediction device 300 proceeds to step S206.

[0079] In step S206, the performance information prediction device 300 extracts the waveform feature data at the reference time from the buffer.

[0080] In step S207, the performance information prediction device 300 inputs the extracted waveform feature data into the performance model.

[0081] In step S208, the performance information prediction device 300 judges whether or not sound information including a playing style and a note number has been output from the performance model. If sound information has been output (S208: Yes), the performance information prediction device 300 assigns the output playing style and note number to variables v and k, respectively, in step S209. On the other hand, if sound information has not been output (S208: No), the performance information prediction device 300 proceeds to step S215.

[0082] In step S210, the performance information prediction device 300 determines whether or not a sound has been produced. If a sound has been produced (S210: Yes), the performance information prediction device 300 generates mute information for the note number K0 of the previous sound production event in step S211. On the other hand, if a sound has not been produced (S210: No), the performance information prediction device 300 proceeds to step S212.

[0083] In step S212, the performance information prediction device 300 generates sound production information including the velocity converted from the rendition style number v, the note number k, and the volume I.

[0084] In step S213, the performance information prediction device 300 assigns k to the note number K0 of the previous sound generation event.

[0085] In step S214, the performance information prediction device 300 identifies a pitch corresponding to the previously occurring pitch P0=k.

[0086] In step S215, the performance information prediction device 300 executes pitch detection on the waveform feature data at the reference time, and stores the detected pitch in p.

[0087] In step S216, the performance information prediction device 300 judges whether p=P0. If p=P0 (S216: Yes), the performance information prediction device 300 proceeds to step S218, and if p=P0 is not satisfied (S216: No), in step S217, the performance information prediction device 300 generates pitch bend information based on the difference from p.

[0088] In step S218, the performance information prediction device 300 increments the string number s by 1 to execute the above-mentioned process for the next string.

[0089] In step S219, the performance information prediction device 300 determines whether all strings have been processed. If all strings have been processed (S219: No), the performance information prediction process ends. If not (S219: Yes), the performance information prediction device 300 repeats the above-described process for the next string.

[0090] In the above embodiment, a performance information prediction system is described in which a performance model that predicts performance information from string vibration waveform data of a stringed instrument such as the guitar 50 is trained, and the performance information is predicted using the trained performance model, but the present disclosure is not limited to this and may be applied to a wind instrument. That is, the present disclosure may be applied to a performance information prediction system in which a performance model that predicts performance information from air vibration waveform data of a wind instrument is trained, and the performance information is predicted using the trained performance model.

[0091] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the specific embodiments described above, and various modifications and variations are possible within the scope of the gist of the present invention as defined in the claims.

[0092] The invention as originally claimed in the present application is set forth below. [Note] In one aspect of the present disclosure, a pre-processing unit that generates waveform characterization data including zero crossing points, minimum points, and maximum points from string vibration waveform data representing a stringed instrument performance; a performance information prediction unit that predicts performance information of the stringed instrument performance from the waveform feature data by using a trained performance model; The present invention provides a performance information prediction device having the above structure.

[0093] In one embodiment, the performance information may consist of sound generation information, sound muting information and pitch modification information.

[0094] In one embodiment, the sound information may include a playing style and a note number.

[0095] In one embodiment, the trained performance model may output the playing style and the note number.

[0096] In one embodiment, the pre-processing unit may perform high-pass filtering and low-pass filtering on the string vibration waveform data.

[0097] In one embodiment, the trained performance model may be implemented as a neural network.

[0098] In one embodiment, the performance information may be described according to the MIDI protocol.

[0099] In another aspect of the present disclosure, a pre-processing unit that generates waveform characterization data including zero crossing points, minimum points, and maximum points from string vibration waveform data representing a stringed instrument performance performed in accordance with training performance information; a performance model training unit that uses the training performance information to train a performance model that predicts performance information of the stringed instrument performance from the waveform characterization data; A performance model training device having the above structure is provided.

[0100] In another aspect of the present disclosure, Electronic string instruments, The performance information prediction device described above; The above-mentioned performance model training device; A performance information generating system having the above structure is provided.

[0101] In another aspect of the present disclosure, generating, by one or more processors, waveform characterization data including zero crossings, minimum points, and maximum points from string vibration waveform data representing a performance of a stringed instrument; predicting, by the one or more processors, performance information of the stringed instrument performance from the waveform characterization data using a trained performance model; A performance information prediction method is provided, which has the following steps:

[0102] In another aspect of the present disclosure, generating, by the one or more processors, waveform characterization data including zero crossing points, minimum points, and maximum points from string vibration waveform data representing a stringed instrument performance performed according to the training performance information; training, by the one or more processors, a performance model that utilizes the training performance information to predict performance information of the stringed instrument performance from the waveform characterization data; A performance model training method is provided, which has the following: [Explanation of symbols]

[0103] 10 Guitar Controller 50 Guitar 100 Control device 200 Performance model training device 210 Pretreatment section 220 Performance Model Training Department 300 Performance Information Prediction Device 310 Pretreatment section 320 Performance Information Prediction Unit

Claims

1. a pre-processing unit that generates waveform characterization data including zero crossing points, minimum points, and maximum points from string vibration waveform data representing a stringed instrument performance; a performance information prediction unit that predicts performance information of the stringed instrument performance from the waveform feature data by using a trained performance model that outputs a playing style and a note number of the stringed instrument from the waveform feature data; A performance information prediction device having the above configuration.

2. 2. The performance information prediction device according to claim 1, wherein the performance information includes sound generation information, sound mute information, and pitch change information.

3. 3. The performance information prediction device according to claim 2, wherein the sound generation information includes the playing style and the note number.

4. 4. The performance information prediction device according to claim 1, wherein the preprocessing unit performs high-pass filtering and low-pass filtering on the string vibration waveform data.

5. 5. The apparatus according to claim 1, wherein the trained performance model is realized by a neural network.

6. 6. The performance information prediction device according to claim 1, wherein the performance information is described in accordance with a MIDI protocol.

7. a pre-processing unit that generates waveform characterization data including zero crossing points, minimum points, and maximum points from string vibration waveform data representing a stringed instrument performance performed in accordance with training performance information; a performance model training unit that uses the training performance information to train a performance model that predicts performance information of the stringed instrument performance from the waveform characterization data; A performance model training device having the above structure.

8. Electronic string instruments, The performance information prediction device according to any one of claims 1 to 6, A musical performance model training device according to claim 7, A performance information generating system having the above configuration.

9. generating, by one or more processors, waveform characterization data including zero crossings, minimum points, and maximum points from string vibration waveform data representing a performance of a stringed instrument; predicting performance information of the stringed instrument performance from the waveform characterization data using a trained performance model that outputs a stringed instrument playing style and note number from the waveform characterization data; The performance information prediction method includes the steps of:

10. generating, by the one or more processors, waveform characterization data including zero crossing points, minimum points, and maximum points from string vibration waveform data representing a stringed instrument performance performed in accordance with the training performance information; training, by the one or more processors, a performance model that utilizes the training performance information to predict performance information of the stringed instrument performance from the waveform characterization data; A performance model training method having the above structure.

Citation Information

Patent Citations

  • Input controller for electronic musical instrument

    JP1988261391A

  • Detection method of musical performance position and detection method of pitch

    JP1997006339A

  • Signal analyzer with at least one strung string and pickup

    JP1997510794A

  • Chord vibration pitch detecting device and chord vibration pitch detecting method

    JP2000105590A

  • Musical instrument performance training device and program therefor

    JP2006030692A