Device, System, Method, and Program

Abstract

To achieve a suitable adjustment of trade-off between jitter and latency.SOLUTION: An electronic device according to an aspect of the present disclosure comprises: a receiving unit that receives a Musical Instrument Digital Interface (MIDI(R)) message, time stamp information regarding execution timing of the MIDI message, and offset information regarding buffering of the MIDI message; and a control unit that controls execution timing of the MIDI message by the time stamp information, based on the offset information. The control unit determines whether or not to execute control of the execution timing by the time stamp information, based on one of system-exclusive messages, control change messages, and assignable controller messages of MIDI.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present disclosure relates to a vice, a system, a method, and a program that utilize wireless communication. De

Background Art

[0002] Musical Instrument Digital Interface (MIDI (registered trademark)) is widely used as a format for exchanging data for playing electronic musical instruments between devices. On the other hand, in recent years when wireless communication devices are widely used, technologies for transmitting and receiving MIDI wirelessly have been studied. For example, MIDI over Bluetooth LE (or BLE MIDI), which transmits and receives MIDI using Bluetooth (registered trademark) Low Energy (Bluetooth LE), has been studied.

[0003] In addition, a dongle device has been proposed that can realize BLE MIDI simply by inserting it into the MIDI terminal of an existing electronic musical instrument (Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Since MIDI data itself does not have time-related data, the timing at which a device receives the data becomes the timing for controlling the musical instrument (such as sound production).

[0006] ​However, if the wireless communication environment is poor (for example, the communication quality is poor), the time until the transmitted data arrives may vary. Also, Bluetooth LE is designed for low power consumption, and data communication is intermittent. Since the timing difference between multiple intermittent transmissions is recommended to be 15 ms (milliseconds) or less, even in a good wireless environment, a data delay (fluctuation) of 15 ms may occur.

[0007] In BLE MIDI and MIDI2.0, timestamps can be used. Regardless of the received timing, if synchronization is achieved with the timestamp, the fluctuation (jitter) can be improved. However, in order to obtain a sufficient improvement effect, it is necessary to increase the data buffer size of the receiving device. In that case, there is a problem that the quantitative delay (latency) increases and the real-time performance is impaired. If the trade-off between jitter and latency cannot be achieved well, the user's perceived quality will decrease, but such a method has not been studied.

[0008] Therefore, one of the objectives of the present disclosure is to provide a vice, system, method, and program that can achieve a suitable adjustment of the trade-off between jitter and latency. De One of the purposes is to provide a vice, system, method, and program.

Means for Solving the Problem

[0009] A device according to an aspect of the present disclosure includes a control unit that determines offset information regarding buffering of Musical Instrument Digital Interface (MIDI (registered trademark)) messages, the MIDI messages, timestamp information regarding the execution timing of the MIDI messages, Timestamp processing timestamp processing information regarding the presence or absence of, and the offset information, and a transmission unit that transmits the above. The timestamp processing information is included in any one of MIDI system exclusive messages, control change messages, and assignable controller messages.

Effect of the Invention

[0010] According to one aspect of the present disclosure, a suitable adjustment of the trade-off between jitter and latency can be achieved.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following description, the same parts are denoted by the same reference numerals. Since the same parts have the same name, function, etc., detailed descriptions will not be repeated.

[0013] (System) FIG. 1 is a diagram showing an example of the schematic configuration of a system according to an embodiment. The system 1 shown in FIG. 1 includes an electronic musical instrument 10, a dongle 20, and a device 30. The system 1 may be referred to as a MIDI transmission (or playback or execution) system, a wireless performance evaluation system, an electronic musical instrument system, or the like.

[0014] The electronic musical instrument 10 is a device that receives an input from a user via an operator such as a keyboard or a switch and controls performances and the like. The electronic musical instrument 10 may be a device having a function of generating a sound according to performance information such as MIDI data. The device may be an electronic musical instrument (such as an electronic piano or a synthesizer), or an analog musical instrument configured to have a function equivalent to that of an electronic musical instrument by mounting a sensor or the like, or an electronic device having no operator (such as a keyboard) for performance (the same electronic device as the device 30 described later). In the present disclosure, the electronic device may mean a name including the electronic musical instrument 10 and other electronic devices. That is, in the present disclosure, the electronic musical instrument 10 may be read interchangeably with the electronic device 10.

[0015] The dongle 20 is connected to the electronic musical instrument 10 and relays direct communication with the device 30. In the present disclosure, the dongle 20 will be described assuming that it is connected (attached) to a Universal Serial Bus (USB) terminal of the electronic musical instrument 10, but the present disclosure is not limited thereto. When an interface other than USB is used, the USB in the present disclosure may be realized by being read in relation to the interface. The dongle 20 may be referred to as a communication device for an electronic musical instrument.

[0016] The device 30 is an electronic device that communicates with the electronic musical instrument 10 (or the dongle 20). The device 30 may be a mobile terminal (mobile communication terminal) such as a mobile phone, a smartphone, or a tablet-type terminal, or a fixed communication terminal such as a personal computer (PC), a server, a television, or a game machine. That is, the device 30 in the present disclosure may be read interchangeably with a communication device, a communication apparatus, a terminal device, or the like.

[0017] <Electronic musical instrument> FIG. 2 is a diagram showing an example of the appearance of the electronic musical instrument 10 according to one embodiment. The electronic musical instrument 10 may be equipped with a switch (button) panel 140b, a keyboard 140k, a display 150d, a speaker 150s, and the like.

[0018] The switch panel 140b may include switches for operating volume designation, sound source, tone color setting, song (accompaniment) selection, song playback start / stop, song playback setting (such as tempo), and the like.

[0019] The keyboard 140k may have a plurality of keys as performance operators. The keys may be called performance operators, pitch operators, tone color operators, direct operators, and the like.

[0020] The display 150d may display lyrics, musical scores, various setting information, and the like. The speaker 150s may be used to emit the sound generated by the performance.

[0021] Note that the electronic musical instrument 10 may be able to generate or convert at least one of MIDI messages (events) and Open Sound Control (OSC) messages. Also, "MIDI" in the present disclosure may be any of the MIDI 1.0 standard, the MIDI 2.0 standard, and standards obtained by modifying / extending MIDI. Further, in the present disclosure, MIDI messages and MIDI data may be read interchangeably. Note that a MIDI message may mean a plurality of (for example, several bytes) MIDI data that function as one command.

[0022] The electronic musical instrument 10 may communicate with a network (such as the Internet) via at least one of wired and wireless (for example, Long Term Evolution (LTE), 5th generation mobile communication system New Radio (5G NR), Wi-Fi (registered trademark), Bluetooth, etc.).

[0023] FIG. 3 is a diagram showing an example of the hardware configuration of a control system 100 of an electronic musical instrument 10 according to an embodiment.

[0024] A central processing unit (CPU) 101, a ROM (read-only memory) 102, a RAM (random access memory) 103, a sound source 104, a key scanner 106 to which the switch (button) panel 140b and the keyboard 140k in FIG. 2 are connected, a USB interface 107, and an LCD controller 108 to which an LCD as an example of the display 150d in FIG. 2 is connected are connected to a system bus 109, respectively.

[0025] A timer 110 (which may be called a counter) for controlling performance may be connected to the CPU 101. The timer 110 may be used, for example, to count the progress of automatic performance in the electronic musical instrument 10. The CPU 101 may be called a processor and may include an interface with peripheral circuits, a control circuit, an arithmetic circuit, registers, and the like.

[0026] The CPU 101 executes the control operation of the electronic musical instrument 10 in FIG. 2 by executing a control program stored in the ROM 102 while using the RAM 103 as a work memory. In addition to the above control program and various fixed data, the ROM 102 may store song data, accompaniment data, song data including these, and the like.

[0027] The key scanner (scanner) 106 constantly scans the key-pressing / releasing state of the keyboard 140k in FIG. 2, the switch operation state of the switch panel 140b, and the like, and interrupts the CPU 101 to convey the state change.

[0028] The LCD controller 108 is an IC (integrated circuit) that controls the display state of an LCD as an example of the display 150d.

[0029] Based on the note-on / off data input from the CPU 101 according to the key scanner 106, the sound source 104 generates a digital sound source signal (for example, waveform data of a musical instrument sound) corresponding to the sound to be emitted (note-on), and outputs it to the Digital to Analog (D / A) converter 111. The sound source 104 may execute processes such as envelope control of the sound to be emitted. The sound source 104 may execute voice synthesis processing to generate a synthesized voice signal.

[0030] The D / A converter 111 converts the input digital signal into an analog signal and outputs it to the amplifier 112. The amplifier 112 may amplify the input signal and output it from the speaker 150s or an output terminal (not shown).

[0031] The USB interface 107 exchanges signals conforming to the USB standard with the dongle 20 physically connected to the electronic musical instrument 10 using an external connection terminal (for example, a connector). The USB interface 107 may include a USB connector (USB connection terminal).

[0032] When the signal input from the USB interface 107 includes data in MIDI format (MIDI data), the CPU 101 may perform playback processing using the sound source 104. Further, the CPU 101 may generate MIDI data based on key input information (for example, note-on / off) acquired from the key scanner 106, etc., output it to the USB interface 107, and transmit it to the device 30 via the dongle 20.

[0033] <Dongle> FIG. 4 is a diagram showing an example of the hardware configuration of the dongle 20 according to an embodiment. The dongle 20 includes, for example, a USB interface unit 201, an RF unit 202, an antenna unit 203, etc. It should be understood by those skilled in the art that each drawing of the present disclosure only shows the configuration used for explanation, and may include configurations such as a power supply not shown.

[0034] The USB interface unit 201 relays signals (for example, signals including MIDI data) conforming to the USB standard between the USB interface 107 of the electronic musical instrument 10 and the RF unit 202. In other words, the USB interface unit 201 has a function of bridging (converting) between the signals used in the RF unit and the signals conforming to the USB standard.

[0035] For example, the USB interface unit 201 acquires the original data from the signals (packets) transmitted via the USB interface 107 of the electronic musical instrument 10 and transfers it to the RF unit 202. Also, the USB interface unit 201 includes the data transferred from the RF unit 202 in a USB packet and transmits it to the USB interface 107 of the electronic musical instrument 10.

[0036] The RF unit 202 realizes transmission and reception of signals using wireless communication (for example, Bluetooth, Wi-Fi, etc.).

[0037] The RF unit 202 may perform transmission processing such as channel coding (which may include error correction coding), modulation, mapping, filtering, inverse fast Fourier transform (IFFT) processing, digital-to-analog conversion, etc. on the bit string to be transmitted (for example, MIDI data) transferred from the USB interface unit 201, and output a baseband signal. The RF unit 202 may perform modulation to a radio frequency band, filtering, amplification, etc. on the baseband signal, and transmit the signal in the radio frequency band via the antenna unit 203.

[0038] On the one hand, the RF unit 202 may perform amplification, filtering, demodulation to a baseband signal, etc. on the signal in the radio frequency band received by the antenna unit 203. The RF unit 2002 may perform reception processing such as analog-to-digital conversion, Fast Fourier Transform (FFT) processing, filtering, demapping, demodulation, decoding (which may include error correction decoding) on the acquired baseband signal, obtain the transmitted data (for example, MIDI data), and transfer it to the USB interface unit 201.

[0039] Note that the RF unit 202 may simultaneously perform a process of converting the received radio signal into the transmitted data and a process of converting the bit sequence to be transmitted into a radio signal.

[0040] The antenna unit 203 can be composed of at least one antenna described based on the common recognition in the technical field related to the present disclosure, for example, a pattern antenna, a chip antenna, a dipole antenna, an omnidirectional antenna, a whip antenna, etc.

[0041] Note that in the present disclosure, other data (for example, audio data about the accompaniment of a piece of music) may be transmitted and received in the USB interface or wireless communication simultaneously with MIDI. Also, the USB interface unit 201 of the present disclosure may be read as any interface unit capable of connecting and communicating with the electronic musical instrument 10. In this case, "USB" in the present disclosure may be read as a name related to any interface unit, and "a signal conforming to the USB standard" may be read as a signal conforming to the standard to which the arbitrary interface unit conforms.

[0042] <Device> FIG. 5 is a diagram showing an example of the hardware configuration of a device 30 according to an embodiment. Physically, the device 30 may be configured as a computer device including a processor 301, a memory 302, a storage 303, a communication device 304, an input device 305, an output device 306, a bus 307, etc.

[0043] Each function in device 30 is realized by causing a processor 301 to perform operations by loading a predetermined software (program) onto hardware such as the processor 301 and the memory 302, and controlling communication by the communication device 304 and reading and / or writing data in the memory 302 and the storage 303.

[0044] The processor 301 controls the entire computer by operating, for example, an operating system. The processor 301 may be constituted by a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic device, a register, and the like.

[0045] Also, the processor 301 reads a program (program code), a software module, data, etc. from at least one of the storage 303 and the communication device 304 into the memory 302, and executes various processes according to these. As the program, a program for causing a computer to execute at least a part of the operations described in the embodiments of the present disclosure is used.

[0046] The memory 302 is a computer-readable recording medium, and may be constituted by, for example, at least one of a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically EPROM), a RAM (Random Access Memory), and other appropriate storage media. The memory 302 may be referred to as a register, a cache, a main memory (main storage device), and the like. The memory 302 can store a program (program code), a software module, etc. executable for implementing the method according to one embodiment.

[0047] Storage 303 is a computer-readable recording medium and may be composed of, for example, at least one of a flexible disk, a floppy (registered trademark) disk, a magneto-optical disk (such as a compact disc (CD-ROM (Compact Disc ROM), etc.), a digital versatile disc, a Blu-ray (registered trademark) disc), a removable disk, a hard disk drive, a smart card, a flash memory device (such as a card, a stick, a key drive), a magnetic stripe, a database, a server, and other appropriate storage media. Storage 303 may be referred to as an auxiliary storage device.

[0048] Communication device 304 is hardware (a transmission / reception device) for performing communication between computers via at least one of a wired network and a wireless network, and is also referred to as, for example, a network device, a network controller, a network card, a communication module, etc.

[0049] Input device 305 is an input device (such as a keyboard, a mouse, etc.) for receiving external input. Output device 306 is an output device (such as a display, a speaker, etc.) for performing external output. Note that input device 305 and output device 306 may have an integrated configuration (such as a touch panel).

[0050] Also, each device such as processor 301 and memory 302 is connected by a bus 307 for communicating information. Bus 307 may be composed of a single bus or may be composed of different buses between devices.

[0051] Note that these system configurations and device configurations are just examples and are not limited thereto. For example, the number of each circuit included is not limited to this. Each device may have a configuration that does not include some circuits (mechanisms), or may have a configuration in which the function of one circuit is realized by a plurality of circuits. It may also have a configuration in which the functions of a plurality of circuits are realized by one circuit.

[0052] Also, the electronic musical instrument 10, the dongle 20, and the device 30 may be configured to include hardware such as a microprocessor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Programmable Logic Device (PLD), and a Field Programmable Gate Array (FPGA), and some or all of each functional block may be realized by the hardware. For example, the CPU 101 may be implemented by at least one of these hardware components.

[0053] Note that the block diagrams described so far show blocks in terms of functions. These functional blocks (components) may be realized by any combination of hardware and / or software. Also, the means for realizing each functional block is not particularly limited. That is, each functional block may be realized by a single physically coupled device, or may be realized by two or more physically separated devices connected by wire or wirelessly.

[0054] For example, although only one processor 301 is shown, there may be a plurality of processors. Also, the processing may be executed by one processor, or the processing may be executed by two or more processors simultaneously, sequentially, or using other methods. Note that each functional block may be implemented by one or more chips.

[0055] (Latency and Jitter Control Method) The latency and jitter control method according to an embodiment of the present disclosure will be described below.

[0056] Latency and jitter are in a trade-off relationship. The inventors have found that although data control is performed on the receiving side (the electronic musical instrument 10 side), the optimal setting value of the receiving-side buffer largely depends on how the application software on the transmitting side (the device 30 side) handles MIDI data, the data configuration of the MIDI data, the performance of the device 30, etc., which is mostly on the transmitting side.

[0057] Therefore, the inventors conceived a control method in which the application software of the device 30 can remotely and real-time set the timestamp control method of the electronic musical instrument 10 and the optimal offset value for buffering to the electronic musical instrument 10 using messages that can be extended / set by the manufacturer in the MIDI data format. According to this configuration, the optimization of the balance between jitter and latency can be achieved. Also, the application software of the device 30 can adjust the optimal balance between latency and jitter without the user being aware of it.

[0058] In this embodiment, a method using a system exclusive message is disclosed as an extendable message, but NRPN (Non Registered Parameter Number) included in the control change message or an assignable controller message of the MIDI2.0 standard may also be used. That is, the system exclusive message in this disclosure may be mutually read as the control change message, NRPN, assignable controller message, etc.

[0059] FIG. 6 is a diagram showing an example of a flowchart of a jitter control method according to an embodiment. In the description of this disclosure, the device 30 may be mutually read as "the application of the device 30".

[0060] Device 30 determines an offset time related to the buffering of MIDI data (step S101). The device 30 may set the offset time to a preset value, a value input by the user, or determine it based on the measurement described later. For example, if the device 30 determines based on the measurement that the jitter (or delay) between the device 30 and the electronic musical instrument 10 is relatively small (or almost 0), the device 30 may determine the offset time to be a relatively small value (or 0).

[0061] Note that the offset time may be interchangeably read as an offset amount, an offset, etc. Hereinafter, the offset time is simply referred to as an offset. The offset may also be referred to as the time related to the buffering of MIDI data.

[0062] Also, in step S101, the device 30 may determine the presence or absence of the timestamp processing of the electronic musical instrument 10. For example, if the device 30 determines based on the measurement that the jitter (or delay) between the device 30 and the electronic musical instrument 10 is relatively small (or almost 0), the device 30 may determine not to apply the timestamp processing, and if not, may determine to apply the timestamp processing.

[0063] The device 30 transmits information about the determined offset to the electronic musical instrument 10 (step S102). This information about the offset (which may also be called offset information) may explicitly indicate the offset time itself (e.g., 15 ms, 30 ms, etc.), or may indicate an index value associated with the offset time. The correspondence between this index value and the offset time may be predefined or set. The correspondence may indicate, for example, index = 0 means offset time = 0 (or no timestamp processing), index = 1 means offset time = 15 ms, etc.

[0064] The electronic musical instrument 10 sets an offset based on the received offset information (step S103). If the electronic musical instrument 10 does not receive the offset information, it may determine that the offset is a predetermined value (default value, for example, 0).

[0065] Thereafter, the device 30 transmits the MIDI data to the electronic musical instrument 10 via wireless communication (for example, using BLE) (step S104). The electronic musical instrument 10 buffers the received MIDI data based on the offset set in step S103, and performs execution (for example, playback, sound generation) processing of the MIDI data (step S105). The execution processing in step S105 may include processing of a note-on message for starting sound generation, processing of a note-off message for ending sound generation, processing of a message for turning a pedal ON / OFF, and other processing related to arbitrary MIDI data.

[0066] In step S102, information regarding the presence or absence of timestamp processing (which may be referred to as timestamp processing information) may be transmitted instead of or together with the offset information.

[0067] In BLE-MIDI and MIDI2.0, a timestamp (time information) may be added when transmitting the original MIDI data. When the timestamp processing information indicates "yes", the electronic musical instrument 10 performs correction based on the timestamp in the execution processing of step S105. Otherwise, even if the received MIDI data has a timestamp, no correction based on the timestamp is performed (the timestamp is ignored and execution is carried out).

[0068] The time stamp may be referred to as information regarding the execution timing of MIDI messages. In the present disclosure, the time stamp and the time stamp information may be read interchangeably. Note that the time stamp may be included in the MIDI data, or may be notified to the electronic musical instrument 10 as information separate from the MIDI data. In the latter case, even when receiving MIDI 1.0 data in which the time stamp information is not included in the standard, control of the time stamp can be applied based on the present disclosure.

[0069] Figures 7A - 7D are conceptual diagrams of corrections based on time stamps and offsets.

[0070] Figure 7A shows an example of the sounding timing as originally performed. In the figure, the notes are arranged in the order of note 0 with the length of an eighth note, note 1 with the length of a quarter note, and note 2 with the length of an eighth note. ts0, ts1, and ts2 correspond to the time stamps indicating the sounding times of note 0, note 1, and note 2, respectively. It shows that note 1 sounded at the time ts1 - ts0 after note 0, and note 2 sounded at the time ts2 - ts0 after note 0.

[0071] In Figures 7B - 7D, the data of Figure 7A is transmitted, and the sounding timing when jitter occurs is shown. st0, st1, and st2 described above each figure indicate the arrival times of notes 0 - 2 at the receiving side. Each timing is affected by jitter, st1 - st0 ≠ ts1 - ts0, and st2 - st0 ≠ ts2 - ts0. Also, the positions of the notes described below each figure indicate the actual sounding times.

[0072] Figure 7B shows an example of the sounding timing when correction based on the time stamp is not performed. In this case, since each note sounds at the arrival time, the intervals of ts0, ts1, and ts2 for the sounding times of notes 0 - 2 are not maintained.

[0073] Figure 7C shows an example of pronunciation timing when performing correction based on timestamps without applying an offset. In this case, the pronunciation timing of the notes that arrived earlier than the reference is corrected. This reference may be, for example, the pronunciation timing based on a timestamp with the arrival time of a certain note as the reference, or the pronunciation timing based on a timestamp with a certain time (e.g., the start time of a piece or performance (such as MIDI playback)) as the reference.

[0074] In the case of Figure 7C, for note 2, since the difference in arrival time (st2 - st0) with respect to note 0 is smaller than the difference in timestamps (ts2 - ts0) (that is, note 2 arrived earlier than the scheduled pronunciation timing), it is corrected to the original pronunciation timing (the pronunciation timing is adjusted from st0 to after the time ts2 - ts0).

[0075] On the other hand, for note 1, since the difference in arrival time (st1 - st0) with respect to note 0 is larger than the difference in timestamps (ts1 - ts0) (that is, note 1 arrived later than the scheduled pronunciation timing), it arrives without being corrected and is pronounced immediately.

[0076] In the case of Figure 7C, for the pronunciation timing of some notes, the intervals between ts0, ts1, and ts2 are not maintained.

[0077] Figure 7D shows an example of pronunciation timing when performing correction based on timestamps with an offset applied. In this case, the pronunciation timing of the notes that arrived earlier than the reference is corrected. Also, the pronunciation timing of the notes that arrived with a delay within the offset with respect to the reference is corrected. This reference may be, as in Figure 9C, the pronunciation timing based on a timestamp with the arrival time of a certain note as the reference, or the pronunciation timing based on a timestamp with a certain time (e.g., the start time of a piece or performance (such as MIDI playback)) as the reference.

[0078] In the case of FIG. 7D, note 0 is sounded after waiting for the offset time after arriving at st0. Note 1 has arrived at st1, but is not sounded immediately, and is sounded after waiting for the time of the difference in timestamps (ts1 - ts0) from the sounding timing of note 0. Note 2 has arrived at st2, but is not sounded immediately, and is sounded after waiting for the time of the difference in timestamps (ts2 - ts0) from the sounding timing of note 0.

[0079] In the case of FIG. 7D, regarding the sounding timing of the notes, the intervals of ts0, ts1, and ts2 are maintained.

[0080] <Format of Offset Information and Timestamp Processing Information> At least one of the offset information and the timestamp processing information transmitted in step S102 described above may be notified to the electronic musical instrument 10 using, for example, a MIDI System Exclusive (SysEx) message, an NRPN (Non Registered Parameter Number) included in a control change message, or the like.

[0081] Both the offset information and the timestamp processing information may be notified by one SysEx message (or NRPN), or the offset information and the timestamp processing information may be notified separately by a plurality of SysEx messages (or NRPN).

[0082] When the offset information is notified by a SysEx message, the SysEx message may include information (bit string) indicating that it includes the offset information and information (bit string) indicating the content of the offset information (for example, the offset time itself).

[0083] When the timestamp processing information is notified by a SysEx message, the SysEx message may include information (bit string) indicating that it includes the timestamp processing information and information (bit string) indicating the content of the timestamp processing information (for example, performing / not performing timestamp processing).

[0084] These pieces of information may be notified, for example, using MIDI data bytes (e.g., data bytes within a SysEx message). In the present disclosure, when any information is notified using MIDI data bytes, it is necessary to configure the information considering that the most significant bit (MSB) of the data byte is 0. That is, the device 30 on the transmission side allocates the information to each data byte in 7-bit units and transmits it, and the electronic musical instrument 10 on the reception side excludes the MSB of each received data byte and combines them to obtain the original information.

[0085] For example, when the offset information to be transmitted is 2 bytes and is "0000000011111111", the device 30 first divides it into 7-bit units to obtain three bit strings: "0000000", "0111111", and "11". Then, the device 30 adds '0' to the beginning of each bit string (and applies bit padding if the bit string is less than 1 byte), obtains three data bytes: "00000000", "00111111", and "01100000", and may include them in a SysEx message and transmit them.

[0086] Note that the number of bits of the offset information / time stamp processing information to be notified may be predefined or may be notified from the device 30. For example, the offset information may be expressed in 4, 8, 16, 32 bits, etc., and the time stamp processing information may be expressed in 1 bit.

[0087] When notifying offset information with NRPN assigned to control numbers 98 and 99 of a control change message, set numerical values indicating that the information is offset information in the MSB and LSB of the NRPN (which may be arbitrarily set by the manufacturer), and the content of the offset information (for example, the offset time itself) may be set in the data entry of the NRPN and transmitted. The timestamp processing information may also be transmitted with NRPN. The offset information and timestamp processing information may be transmitted by an assignable controller message newly defined as a message corresponding to NRPN in MIDI 2.0.

[0088] (Wireless Performance Measurement Method) Depending on the environment around the electronic musical instrument 10 and the device 30, the wireless communication quality between them may vary, which may affect jitter and latency. For this reason, it is preferable that the wireless performance measurement between the electronic musical instrument 10 and the device 30 can be performed easily and in real time.

[0089] The inventors of the present invention conceived a data format and a measurement method therefor.

[0090] FIG. 8 is a sequence diagram of a wireless performance measurement method according to an embodiment.

[0091] The device 30 transmits measurement data to the electronic musical instrument 10 (step S201). The transmission in step S201 may be a single transmission or a plurality of transmissions at a certain transmission interval (for example, 7.5 ms, 15 ms, etc.) over a certain period (for example, 1 s).

[0092] The electronic musical instrument 10 corrects the counter value for each of the received measurement data and returns it to the device 30 (step S202). The corrected measurement data may be called return data, reply data, report data, etc.

[0093] Based on the transmission data volume, transmission time of the measurement data in step S201, reception time of the returned data in step S202, etc., the device 30 calculates the wireless performance between the electronic musical instrument 10 and the device 30 (step S203). The wireless performance may correspond to at least one of, for example, throughput per unit time, round-trip delay time, etc.

[0094] In this example, the device 30 is described as the transmission side and the electronic musical instrument 10 as the reception side, but the electronic musical instrument 10 may be the transmission side and the device 30 the reception side.

[0095] Before step S201, information for communicating the start of measurement may be transmitted from one of the device 30 and the electronic musical instrument 10 to the other. The device 30 and the electronic musical instrument 10 may reset the identification number, counter value, etc., and perform time synchronization (time adjustment), etc., described later, triggered by the transmission and reception of the said information.

[0096] Also, the number of transmissions (number of transmitted data), transmission interval, transmission period (measurement period), etc., of the measurement data may be defined in advance in the device 30, or may be determined by the device 30 based on an input from the user. Information regarding the number of transmissions, transmission interval, transmission period, etc., of the measurement data may be notified from the device 30 to the electronic musical instrument 10.

[0097] The device 30 may simultaneously measure the received signal strength (Received Signal Strength Indicator (RSSI)) during this measurement. The said RSSI may be an instantaneous RSSI, or may be an average / minimum / maximum RSSI over a certain period.

[0098] <Format of Measurement Data / Reply Data> The measurement data transmitted in step S201 described above may be notified to the electronic musical instrument 10 using at least one of MIDI SysEx messages, control change messages, etc.

[0099] FIG. 9 is a diagram showing an example of the format of measurement data. The format of this example corresponds to a SysEx message, starts with "F0" indicating the start of the SysEx message, and ends with "F7" indicating the end of the SysEx message. The description of other information necessary for other SysEx messages is omitted.

[0100] When measurement data is notified by a SysEx message, the SysEx message may include information (bit string) of an identification number indicating which data the data is, and information (bit string) of a counter value indicating the value of a certain counter.

[0101] These pieces of information may be notified, for example, using MIDI data bytes (for example, data bytes within a SysEx message).

[0102] For example, the identification number information and the counter value information may be expressed in 4, 8, 16, 32 bits, etc.

[0103] In step S201, when the device 30 transmits the i-th (i is an integer) measurement data among N consecutive (N is an integer) measurement data, as the identification number information, for example, a value of i - 1 is set and transmitted. In other words, for one measurement, the identification number starts from 0, and the device 30 sets and transmits the value incremented by 1 for each data. That is, the identification number may mean "the number of times measurement data has been transmitted - 1". Also, when the device 30 transmits any measurement data, it sets and transmits an arbitrary value (for example, 0) as the counter value.

[0104] In step S202, each time the electronic musical instrument 10 receives measurement data, it increments its own counter value by +1. Note that the electronic musical instrument 10 may set the counter value to -1 at the start of measurement. That is, the counter value of the electronic musical instrument 10 may mean "the number of times measurement data has been received - 1".

[0105] Also, in step S202, the electronic musical instrument 10 may return to the device 30, as return data, the received measurement data with the identification number information unchanged and its own counter value set (updated) in the counter value information.

[0106] That is, the formats of the measurement data and the return data may be the same. In order to suppress the influence on communication due to the difference in data size (improve the measurement accuracy), it is preferable that the sizes (number of bits) of the measurement data and the return data are the same.

[0107] If the identification number and the counter value match in the received return data, the device 30 can determine that there is no loss (correct transmission and reception) in the measurement data transmitted so far.

[0108] The device 30 may derive the round trip time (RTT) between the device 30 and the electronic musical instrument 10 based on the transmission time of the measurement data and the reception time of the return data with the same identification number. Also, the device 30 may derive the communication throughput between the device 30 and the electronic musical instrument 10 based on the communication volume of the measurement data / return data per unit time. The RTT and the communication throughput may be instantaneous values or average / maximum / minimum values over a certain period.

[0109] Also, the receiving side (electronic musical instrument 10) may record the deviation of the measurement data transmitted at equal intervals and calculate the jitter from the device 30 to the electronic musical instrument 10 based on these deviations. The electronic musical instrument 10 may transmit the calculated jitter to the device 30. The electronic musical instrument 10 may transmit information regarding the jitter using a SysEx message.

[0110] In step S202, the electronic musical instrument 10 may set information regarding the reception time instead of the counter value information, using the identification number information of the received measurement data as it is, and reply to the device 30 as return data. It is preferable that the information regarding the reception time has the same number of bits (size) as the counter value information. The device 30 may calculate the jitter from the received return data.

[0111] Note that the electronic musical instrument 10 may also reply to the device 30 with return data including the counter value information and the information regarding the reception time. The electronic musical instrument 10 may also reply to the device 30 with return data including the identification number information, the counter value information, and the information regarding the reception time.

[0112] According to the measurement method as described above, for example, even when other traffic (such as audio data) other than MIDI flows simultaneously with MIDI between the electronic musical instrument 10 and the device 30, the wireless performance measurement can be easily performed, so that the control of jitter and latency in various environments can be suitably performed.

[0113] (Modification example) In each of the above-described embodiments, the dongle 20 may not be provided. In this case, it is sufficient if the electronic musical instrument 10 has a function for wireless communication with the device 30 (for example, the RF unit 202 and the antenna unit 203 in FIG. 4).

[0114] In each of the above-described embodiments (particularly FIGS. 6 and 8, etc.), the operation of the electronic musical instrument 10 may be performed by the dongle 20. For example, the offset in step S103 in FIG. 6 may be set in the dongle 20. In this case, the buffering in step S105 may be performed by the dongle 20, and the electronic musical instrument 10 may receive the MIDI buffered by the dongle 20 for the offset time and immediately perform the execution process of the MIDI data.

[0115] Also, the transmission of the return data in step S202 in FIG. 8 may be performed by the dongle 20. In this case, it is expected that the performance of only the wireless part can be measured more precisely.

[0116] In each of the above embodiments (particularly in FIGS. 6 and 8, etc.), the operation of the electronic musical instrument 10 may be performed by the dongle 20. For example, the offset in step S103 of FIG. 6 may be set in the dongle 20. In this case, the buffering in step S105 is performed by the dongle 20, and the electronic musical instrument 10 may receive the MIDI buffered by the dongle 20 for the offset time and immediately perform the execution process of the MIDI data.

[0117] In the above embodiments, an example where the electronic musical instrument 10 is a keyboard instrument such as a keyboard is shown, but it is not limited thereto. The electronic musical instrument 10 may be any device having a configuration that can specify the sounding timing by a user's operation, such as an electric violin, an electric guitar, a drum, or a trumpet.

[0118] Also, the electronic musical instrument 10 is not limited to a so-called musical instrument (such as a keyboard), and may be replaced with a mobile phone, a smartphone, a tablet terminal, a personal computer (PC), a television, or the like.

[0119] As described above, by using the electronic musical instrument 10, the dongle 20, the device 30, etc. of the present disclosure, it is possible to achieve a suitable adjustment of the trade-off between jitter and latency.

[0120] Note that the electronic device 10 (for example, the electronic musical instrument 10; the same applies hereinafter) may include a receiving unit (for example, the USB interface 107) that receives measurement data including identification number information and counter value information, and a transmitting unit (for example, the USB interface 107) that updates the counter value information of the measurement data based on the number of times the measurement data is received and transmits the data for return. With such a configuration, it is possible to easily perform the measurement of wireless communication for the electronic device.

[0121] Further, the receiving unit may receive a plurality of the measurement data transmitted at a certain transmission interval (for example, 15 ms), and the transmitting unit may transmit the return data for each of the plurality of the measurement data. According to such a configuration, measurement of the throughput, jitter, etc. of the wireless communication in a certain period can be suitably performed.

[0122] Further, the measurement data may be a system exclusive message, a control change message, or an assignable controller message of Musical Instrument Digital Interface (MIDI (registered trademark)). According to such a configuration, measurement suitable for wireless communication (for example, BLE-MIDI) between devices that can use MIDI can be performed.

[0123] Further, the receiving unit may receive the measurement data via a dongle (dongle 20) physically connected to the electronic device 10, and the transmitting unit may transmit the return data via the dongle. According to such a configuration, when the electronic device 10 does not have a function of directly performing wireless communication with the device 30, measurement of the wireless communication between them via the dongle 20 can be realized.

[0124] Note that the electronic device 10 may include a receiving unit that receives a Musical Instrument Digital Interface (MIDI (registered trademark)) message, time stamp information (for example, USB interface 107) regarding the execution timing of the MIDI message, and offset information regarding the buffering of the MIDI message, and a control unit (for example, CPU 101) that controls the execution timing of the MIDI message based on the time stamp information according to the offset information. According to such a configuration, for the wireless communication for the electronic device, a suitable adjustment of the trade-off between jitter and latency can be realized.

[0125] Further, the receiving unit may receive the offset information included in a MIDI system exclusive message, a control change message, or an assignable controller message. According to such a configuration, control suitable for wireless communication (for example, BLE-MIDI) between devices that can use MIDI can be implemented.

[0126] Further, the control unit may determine whether to control the execution timing based on the timestamp information based on a MIDI system exclusive message, a control change message, or an assignable controller message. According to such a configuration, control suitable for wireless communication (for example, BLE-MIDI) between devices that can use MIDI can be implemented.

[0127] Further, the receiving unit may receive the timestamp information, the offset information, and the MIDI message via a dongle (dongle 20) physically connected to the electronic device 10. According to such a configuration, when the electronic device 10 does not have a function of directly wirelessly communicating with the device 30, a suitable adjustment of the trade-off between jitter and latency for wireless communication between them via the dongle 20 can be realized.

[0128] Each aspect / embodiment described in the present disclosure may be used alone, in combination, or switched and used during execution. Also, the processing procedures, sequences, flowcharts, etc. of each aspect / embodiment described in the present disclosure may be rearranged as long as there is no contradiction. For example, for the method described in the present disclosure, the elements of various steps are presented using an exemplary order and are not limited to the specific order presented.

[0129] The description "based on" used in the present disclosure does not mean "only based on" unless otherwise specified. In other words, the description "based on" means both "only based on" and "at least based on".

[0130] Any reference to an element using terms such as "first", "second", etc. used in this disclosure does not generally limit the quantity or order of those elements. These terms can be used in this disclosure as a convenient way to distinguish between two or more elements. Therefore, the reference to the first and second elements does not mean that only two elements can be employed or that the first element must precede the second element in any form.

[0131] In this disclosure, when terms such as "include", "including" and their variants are used, these terms are intended to be inclusive, similar to the term "comprising". Furthermore, the term "or" used in this disclosure is not intended to be an exclusive disjunction.

[0132] "A / B" in this disclosure may mean "at least one of A and B".

[0133] In this disclosure, for example, when articles are added by translation, such as a, an and the in English, this disclosure may include that the nouns following these articles are in the plural form.

[0134] As described in detail above about the invention according to this disclosure, it is obvious to those skilled in the art that the invention according to this disclosure is not limited to the embodiments described in this disclosure. The invention according to this disclosure can be implemented in modified and changed forms without departing from the spirit and scope of the invention determined based on the description of the claims. Therefore, the description of this disclosure is for illustrative purposes and does not bring any limiting meaning to the invention according to this disclosure.

Claims

1. A control unit that determines offset information regarding buffering of Musical Instrument Digital Interface (MIDI (registered trademark)) messages, A transmission unit that transmits the MIDI message, timestamp information regarding the execution timing of the MIDI message, timestamp processing information regarding the presence or absence of timestamp processing, and the offset information, comprising: The timestamp processing information is a device included in any of a MIDI system exclusive message, a control change message, and an assignable controller message.

2. The device according to claim 1, wherein the transmission unit transmits the offset information included in any of the MIDI system exclusive message, the control change message, and the assignable controller message to an electronic device.

3. The device according to claim 1 or 2, wherein the transmission unit transmits the offset information included in any of the MIDI system exclusive message, the control change message, and the assignable controller message to a dongle physically connected to an electronic device.

4. The device according to any one of claims 1 to 3, wherein when the timestamp processing information is information indicating that there is no timestamp processing, the information does not cause correction based on the timestamp information even if the timestamp information is given.

5. The device according to any one of claims 1 to 4, wherein the transmission unit includes information indicating that the timestamp processing information is included in the MIDI system exclusive message and transmits the same.

6. A system including an electronic device and a device, wherein the device has a control unit that determines offset information regarding buffering of Musical Instrument Digital Interface (MIDI (registered trademark)) messages, and a transmission unit that transmits the MIDI message, timestamp information regarding the execution timing of the MIDI message, timestamp processing information regarding the presence or absence of timestamp processing, and the offset information, comprising: The time stamp processing information is included in any one of a MIDI system exclusive message, a control change message, and an assignable controller message. The electronic device A receiving unit that receives the MIDI message, the time stamp information, and the offset information; A control unit that controls the execution timing of the MIDI message based on the time stamp information according to the offset information, and determines whether to control the execution timing according to the time stamp information based on the time stamp processing information included in any one of the MIDI system exclusive message, the control change message, and the assignable controller message. A system having a control unit.

7. Determining offset information regarding buffering of Musical Instrument Digital Interface (MIDI (registered trademark)) messages; Transmitting the MIDI message, time stamp information regarding the execution timing of the MIDI message, time stamp processing information regarding the presence or absence of time stamp processing, and the offset information. A method of a device, wherein the time stamp processing information is included in any one of a MIDI system exclusive message, a control change message, and an assignable controller message.

8. Causing a computer of a device to Determine offset information regarding buffering of Musical Instrument Digital Interface (MIDI (registered trademark)) messages; Cause the MIDI message, time stamp information regarding the execution timing of the MIDI message, time stamp processing information regarding the presence or absence of time stamp processing, and the offset information to be transmitted; A program, wherein the time stamp processing information is included in any one of a MIDI system exclusive message, a control change message, and an assignable controller message.

Images