Controller, musical sound generation system, method, program, and storage medium

Abstract

To provide a controller which allows a user with limited physical movement to easily control a musical sound generation device, and to provide a musical sound generation system, method, program, and storage medium.SOLUTION: In a musical sound generation system 1, a controller for controlling a musical sound generation device comprises: an acquisition unit for acquiring user line-of-sight information; a display control unit configured to display, on the display unit, multiple operators associated with control commands of the musical sound generation device, and to display, on the display unit, a pointer for selecting an operator according to the line-of-sight information acquired by the acquisition unit; a determination unit configured to determine selection of an operator overlapping with at least a portion of the pointer from among the multiple operators; and a communication unit configured to send, to the musical sound generation device, information compliant with the MIDI standard and corresponding to the operator whose selection was determined by the determination unit.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a controller, a musical sound generation system, a method, a program, and a storage medium. [Background technology]

[0002] BACKGROUND ART Conventionally, a performance interface is known that generates performance control information for controlling musical tones generated from a musical tone generating device in accordance with the physical condition of an operator (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2001-195059 Summary of the Invention [Problem to be solved by the invention]

[0004] However, nothing has been proposed so far regarding use by users with limited physical movement, such as ALS patients.

[0005] The problem to be solved by the present invention is to enable a user with limited physical movement to easily control a musical sound generating device. [Means for solving the problem]

[0006] The controller of the first embodiment is a controller that controls a musical sound generating device, and includes an acquisition unit that acquires a user's gaze information, a display control unit that displays a plurality of operators associated with control commands of the musical sound generating device on a display unit and displays a pointer on the display unit for selecting the operator based on the gaze information acquired by the acquisition unit, a determination unit that determines the selection of an operator from the plurality of operators that overlaps with at least a portion of the pointer, and a communication unit that transmits information compliant with the MIDI standard corresponding to the operator whose selection has been determined by the determination unit to the musical sound generating device.

[0007] This configuration allows a user with limited physical movement to easily control the musical sound generating device.

[0008] The controller according to the second aspect is the controller according to the first aspect, further comprising a reception unit that receives input operations by a user using an input device, and the determination unit determines the selection of the operator in accordance with the input operation received by the reception unit.

[0009] According to this configuration, by using both input using the user's line of sight information (line of sight input) and input using the input device, it is possible to quickly determine the selection of an operator.

[0010] A controller according to a third aspect is the controller according to the second aspect, wherein the input operation is pressing a switch with a part of the user's body, and the determination unit determines the selection of the operator based on the user's line of sight information and without any user operation other than pressing the switch.

[0011] According to this configuration, since the operator can be selected by the user's line of sight information and switch operation, even a user with limited physical movement can easily control the musical sound generating device.

[0012] A controller according to a fourth aspect is the controller according to any one of the first to third aspects, wherein the decision section decides to select an operator that has been overlapping with at least a part of the pointer for a predetermined period of time.

[0013] According to this configuration, even a user who has difficulty moving his or her arms or legs can select an operator by using line-of-sight information.

[0014] A controller according to a fifth aspect is the controller according to any one of the first to fourth aspects, wherein the decision section decides to select an operator that overlaps with the reference position of the pointer.

[0015] This configuration allows for more accurate selection of the controls.

[0016] A controller according to a sixth aspect is the controller according to the fifth aspect, wherein the reference position is approximately the center of the pointer.

[0017] This configuration allows for easier selection of the controls.

[0018] A controller according to a seventh aspect is the controller according to any one of the first to sixth aspects, wherein the display control section displays the beat and bar number of the music being played near the pointer.

[0019] With this configuration, the user can operate the controller while always checking the beats and bar numbers of the music being played.

[0020] A controller according to an eighth aspect is the controller according to the seventh aspect, wherein the display control unit enlarges an area including the pointer and displays the enlarged area on the display unit.

[0021] According to this configuration, a desired operator can be easily selected from among a plurality of operators.

[0022] A controller according to a ninth aspect is the controller according to any one of the first to eighth aspects, wherein the display control section displays more of the plurality of operators near the center of the screen of the display section than at the periphery of the screen.

[0023] According to this configuration, even a user with limited eye movement can easily select an operator.

[0024] A controller according to a tenth aspect is the controller according to any one of the first to ninth aspects, wherein the plurality of controls include an XY pad capable of simultaneously changing two types of parameters.

[0025] According to this configuration, the XY pad can be operated using line-of-sight information.

[0026] A controller according to an eleventh aspect is the controller according to the first aspect, wherein the determination section determines the selection of the operator without any user operation other than line-of-sight information of the user.

[0027] According to this configuration, since an operator can be selected only by the user's line of sight information (line of sight input), even a user with limited physical freedom can easily control the musical sound generating device.

[0028] A musical sound generating system according to a twelfth aspect is a musical sound generating system comprising a musical sound generating device and a controller for controlling the device, and includes an acquisition unit that acquires a user's line of sight information, a display control unit that displays a plurality of operators associated with control commands of the musical sound generating device on a display unit and displays a pointer on the display unit for selecting the operator based on the line of sight information acquired by the acquisition unit, a determination unit that determines which of the plurality of operators overlaps with at least a portion of the pointer, and a communication unit that transmits information conforming to the MIDI standard corresponding to the operator whose selection has been determined by the determination unit to the musical sound generating device.

[0029] A method according to a thirteenth aspect is a method for controlling a musical sound generating device, characterized in that it comprises the steps of acquiring a user's gaze information, displaying a plurality of operators associated with control commands for the musical sound generating device on a display unit, and displaying a pointer for selecting the operator on the display unit based on the gaze information, determining a selection of an operator from the plurality of operators that overlaps with at least a portion of the pointer, and transmitting information conforming to the MIDI standard corresponding to the determined operator to the musical sound generating device.

[0030] This configuration allows a user with limited physical movement to easily control the musical sound generating device.

[0031] A program according to a fourteenth aspect is a program for causing a computer to execute a method for controlling a musical sound generating device, the method comprising the steps of: displaying a plurality of operators associated with control commands for the musical sound generating device on a display unit; and displaying a pointer on the display unit for selecting the operator based on a user's line of sight; determining a selection of an operator from the plurality of operators that overlaps with at least a portion of the pointer; and transmitting information conforming to the MIDI standard corresponding to the determined operator to the musical sound generating device.

[0032] A storage medium according to a fifteenth aspect is a computer-readable storage medium that stores the program according to the fourteenth aspect. [Effects of the Invention]

[0033] According to the present invention, a user with limited physical movement can easily control a musical sound generating device. [Brief explanation of the drawings]

[0034] [Figure 1] 1 is a diagram showing an example of a system configuration of a musical sound generating system 1 according to an embodiment of the present invention. [Figure 2] 1 is a block diagram showing an example of a hardware configuration of a musical sound generating system 1 according to the present embodiment. [Figure 3] 1 is a block diagram showing an example of a functional configuration of a musical sound generating system 1 according to the present embodiment. [Figure 4] 4 is a flowchart showing an example of the operation of the musical sound generating system 1 according to the present embodiment. [Figure 5] FIG. 2 is a diagram showing an example of a user interface of the controller 2 according to the present embodiment. [Figure 6] FIG. 6 is an enlarged view of area A of the user interface shown in FIG. 5. [Figure 7] 10A and 10B are diagrams showing a modified example of the user interface of the controller 2 according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0035] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In each drawing, components having equivalent functions are designated by the same reference numerals, and detailed description of the components having the same reference numerals will not be repeated.

[0036] (Configuration of musical tone generation system) Fig. 1 is a diagram showing a schematic configuration of a musical sound generation system 1 according to an embodiment. As shown in Fig. 1, the musical sound generation system 1 according to this embodiment includes a controller 2 and a musical sound generation device 3. The controller 2 and the musical sound generation device 3 are communicably connected via a communication interface (I / F) described below.

[0037] The controller 2 is a device for controlling the musical sound generating device 3 based on a communication protocol conforming to the MIDI standard. The controller 2 is connected to an eye-tracking device 21 that allows a user with limited physical movement, such as an ALS patient, to input various information to the controller 2 by eye-tracking, and an input device 22 that allows the user to perform manual operations (the controller 2 may be equipped with the eye-tracking device 21 and the input device 22).

[0038] The eye tracking device 21 tracks the movement of the user's eyes (gaze direction) and outputs gaze information to the controller 2. The gaze information can be generated, for example, by extracting characteristic patterns such as pupils from images of the user's eyes captured sequentially. This embodiment focuses on the fact that ALS patients can move their gaze, and controls the musical sound generating device 3 using the gaze information.

[0039] The input device 22 is an input device that can be operated by a user, such as a manual switch or a foot switch. When a user performs an operation using the input device 22, a signal corresponding to the operation is output from the input device 22 to the controller 2. The input device 22 is preferably one that allows a user with limited physical movement, such as an ALS patient, to easily perform input operations, and the input operation is, for example, to press a switch with the user's body.

[0040] The musical sound generating device 3 is a device for generating musical sounds based on signals conforming to the MIDI standard received from the controller 2. For example, a personal computer can be used as a musical sound generating device by installing and running integrated music software called a DAW (Digital Audio Workstation) on the personal computer. The musical sound generating device 3 also includes a sound system 31 for emitting sounds based on acoustic signals.

[0041] (Hardware configuration) Next, a description will be given of the hardware configuration of the musical sound generating system 1 according to this embodiment. Fig. 2 is a block diagram showing an example of the hardware configuration of the musical sound generating system 1 according to this embodiment.

[0042] In the controller 2, the CPU 23 is a processing device that controls the overall operation of the controller 2. The ROM 24 is a non-volatile memory that stores the control programs executed by the CPU 23 and various data. The RAM 25 is a volatile memory that is used as a load area and work area for the programs executed by the CPU 23. The display 26 is a display device that displays various information (user interface). The storage device 27 is storage means for storing various information, and may be built into the controller 2 body or may have a removable storage medium. The communication interface (I / F) 28 is an interface for connecting to the musical sound generating device 3. The bus 29 is a bus line that connects the above components to each other.

[0043] In this embodiment, the musical sound generating device 3 is a PC with DAW software installed. By executing the DAW software, various music production functions (for example, hard disk recording function, MIDI data and audio data creation and editing function, mixing function, sequencer function, etc.) can be realized. Note that, although this embodiment will be described using an example in which the musical sound generating device 3 is configured with a PC and DAW software, it may also be a standalone sequencer device.

[0044] In the musical sound generating device 3, the CPU 32, ROM 33, RAM 34, storage device 36, input device 37 such as a mouse and keyboard, and display 38 are basic components of a general-purpose PC. The communication I / F 35 is an interface for connecting to the controller 2. The communication I / Fs 28 and 35 may be, for example, a wired music I / F such as MIDI, a general-purpose wired I / F such as USB or IEEE1394, or a general-purpose wireless I / F such as wireless LAN or Bluetooth (registered trademark). The sound system 31 emits analog audio signals passed from a codec, which is a sound I / O equipped with a digital-to-analog (D / A) conversion function. This sound system 31 is an added function in the form of a board or card inserted into a slot provided in the PC constituting the musical sound generating device 3. The bus 39 is a bus line that interconnects the above components.

[0045] (Functional configuration) Next, a description will be given of the functional configuration of the musical sound generating system 1 according to this embodiment. Fig. 3 is a diagram showing an example of the functional configuration of the musical sound generating system 1 according to this embodiment.

[0046] As shown in FIG. 3, the controller 2 includes a control unit 200, a display unit 210, a storage unit 211, and a communication unit 212.

[0047] The control unit 200 includes an acquisition unit 201 , a reception unit 202 , a determination unit 204 , and a display control unit 203 .

[0048] The acquisition unit 201 acquires gaze information of the user of the controller 2. Specifically, the acquisition unit 201 acquires information (gaze information) relating to the gaze direction of the user output from the eye tracking device 21.

[0049] The receiving unit 202 receives an input operation by a user using the input device 22 (a manual switch, a foot switch, etc.).

[0050] The display control unit 203 displays, on the display unit 210 (corresponding to the display 26 in FIG. 2), a plurality of operators (icons) associated with the control commands of the musical sound generating device 3 together with various information. Furthermore, the display control unit 203 displays, on the display unit 210, a pointer for operating the operators, based on the user's line of sight information acquired by the acquisition unit 201.

[0051] 5 is a diagram showing an example of a user interface 50 displayed on the display unit 210 by the display control unit 203. As shown in FIG. 5, the user interface 50 includes, as indicators, an index 51 and a performance scene name section 52. The user interface 50 also includes, as a plurality of controls, a play button 53, a stop button 54, effector buttons 55 and 56, synthesizer buttons 57 to 60, an effector type selection button 61, mapping buttons 62 and 63, an XY pad 64, an auto-play button 65, and a manual-play button 66. Each of these controls 53 to 66 is associated with a control command of the musical sound generation device 3 (DAW), and when the user operates (selects) each of the controls 53 to 66, the musical sound generation device 3 performs processing according to the control command.

[0052] The indicator 51 indicates the beat and bar number of the music being played in the format of "000.00 (beat.bar number)." The performance scene name section 52 displays, for example, the name of the user who is the performer.

[0053] The play button 53 is a control for starting playback of a song. The stop button 54 is a control for stopping playback of a song. The effector buttons 55 and 56 are controls for applying audio effects. The synthesizer buttons 57-60 are controls for playing musical scales and synthesizing musical tones. Three types of controls can be selected for each of the controls 55-60, and a number icon (1-3) is displayed next to each control. The effector type selection button 61 is a control for selecting the type of effector. The mapping buttons 62 and 63 are controls for assigning the X- and Y-directional operations of the XY pad 64, which accepts two-dimensional operations, to different increase / decrease controls of the DAW. The XY pad 64 is a control that can simultaneously change two types of parameters. Two types of parameters can be simultaneously changed by moving the pointer P on the XY pad 64. The operation of the XY pad 64 may be enabled while the accepting unit 202 accepts operations from the input device 22. For example, the user may operate the XY pad 64 (change two types of parameters) by moving the pointer P on the XY pad 64 by eye-gaze input while pressing the manual switch (input device 22). The auto-play button 65 is an operator for playing pre-recorded musical sounds. The manual play button 66 is an operator for starting real-time performance and recording.

[0054] The user interface 50 also includes a pointer P for operating / selecting each of the above-mentioned controls. The pointer P is displayed on the display unit 210 based on the line-of-sight information acquired by the acquisition unit 201. Specifically, the position of the point of gaze on the screen of the display unit 210 is obtained from the line-of-sight information, and the pointer P is displayed at that position.

[0055] Fig. 6 is an enlarged view of area A in Fig. 5. As shown in Fig. 6, pointer P has a reference position P1 and a circular area P2 of a predetermined size that is approximately centered at reference position P1. In addition, an indicator Q is displayed near circular area P2 of pointer P. Like indicator 51 described above, indicator Q indicates the beat and bar number of the music being played in the format "000.00 (beat.bar number)."

[0056] 6, the display control unit 203 can also enlarge and display a portion of the user interface 50 (for example, area A). This allows the user to operate / select each operator more reliably. When a portion of the user interface 50 is enlarged and displayed, the indicator 51 displayed in the upper left corner of the screen of the user interface 50 is outside the display range, but since the indicator Q is displayed near the pointer P as described above, even when the display is enlarged, the user can operate each operator while always checking the beat and bar number of the music being played.

[0057] Fig. 7 is a diagram showing a user interface 70 according to a modified example of the user interface 50. As shown in Fig. 7, in the user interface 70, a plurality of operators are displayed (arranged) more near the center of the screen of the display unit 210 than at the periphery of the screen. This allows even a user with limited eye movement who has difficulty moving their line of sight from one end of the screen to the other of the display unit 210 to easily operate the operators.

[0058] Returning to FIG. 3 , the determination unit 204 determines the selection of an operator that overlaps with at least a part of the pointer P from among the plurality of operators. This allows the selection of an operator to be determined solely through gaze input using the eye tracking device 21. The determination unit 204 may also be configured to determine the selection of an operator that overlaps with the reference position P1 of the pointer P. This reduces the risk of erroneous operation (erroneous selection) of an operator when operators are arranged adjacent to each other. The determination unit 204 may also be configured to determine the selection of an operator that has been overlapping with at least a part of the pointer P for a predetermined time (e.g., two seconds). This allows the selection of an operator to be determined more reliably. This selection method allows the selection of an operator to be determined solely through gaze information, and is therefore useful for users who have difficulty performing actions other than moving their gaze.

[0059] Furthermore, the determination unit 204 may determine the selection of an operator in accordance with an input operation by the user using the input device 22, which is received by the receiving unit 202. That is, the user can select an operator by gaze input using the eye tracking device 21, and then determine the selection of the operator using the input device 22 (a manual switch, a foot switch, or the like). This allows the user to determine the selection of an operator more quickly than when determining the selection of an operator that has been overlapping with at least a part of the pointer P for a predetermined period of time. This selection method is useful for users who can operate the input device 22 with parts of their body, such as their hands or feet, in addition to moving their gaze.

[0060] The storage unit 211 stores information relating to each control command of the musical sound generating device 3 (DAW) associated with each operator.

[0061] The communication unit 212 transmits to the musical sound generating device 3 information (control command information, MIDI event information) conforming to the MIDI standard corresponding to the operator selected by the determination unit 204 .

[0062] Next, we will explain the functional configuration of the musical sound generating device 3. As shown in Fig. 3, the musical sound generating device 3 has an audio input unit 301, a sequencer 302, a recorder 303, a sound source 304, a mixer 305, a sound system 306, and a communication unit 307.

[0063] The audio input unit 301 receives input of an analog audio signal converted into a digital form, or inputs a digital audio signal itself.

[0064] The sequencer 302 records the MIDI event information received by the communication unit 307. This MIDI event information is supplied to the sound source 304, which will be described later.

[0065] The sound source 304 is a software sound source. The sound source 304 generates an audio signal in response to input MIDI event information. The audio signal output from the sound source 304 is input to the recorder 303 and the mixer 305.

[0066] The recorder 303 records the audio signal input by the audio input unit 301 and the audio signal output from the sound source 304. The audio signal recorded in the recorder 303 is output to the mixer 305 in response to a playback instruction.

[0067] The mixer 305 supplies the audio signals output from the sound source 304 and the recorder 303 to the sound system 306 .

[0068] (Example of operation) Next, a description will be given of an example of the operation of the musical sound generating system 1. Fig. 4 is a flowchart showing an example of the operation of the musical sound generating system 1.

[0069] First, the acquisition unit 201 of the control unit 200 in the controller 2 acquires the user's gaze information output from the eye tracking device 21 (step S1). The gaze information may be acquired at any timing, for example, constantly in real time.

[0070] Next, the display control section 203 of the control section 200 displays on the display section 210 a plurality of operators associated with the control commands of the musical sound generating device 3 (DAW) (step S2).

[0071] Furthermore, the display control unit 203 of the control unit 200 displays a pointer for operating the operator on the display unit 210 based on the line-of-sight information acquired by the acquisition unit 201 (step S3).

[0072] Next, the determination unit 204 of the control unit 200 determines the selection of an operator that overlaps at least a part of the pointer from among the multiple operators (step S4).

[0073] Next, the communication unit 307 transmits information conforming to the MIDI standard corresponding to the operator selected by the determination unit 204 to the musical sound generating device 3 (step S5).

[0074] Then, the musical sound generating device 3 performs various processes (execution of control commands, recording of MIDI events, etc.) based on the information conforming to the MIDI standard received from the controller 2 (step S6).

[0075] As described above, according to this embodiment, the musical sound generating system 1 includes an acquisition unit 201 that acquires the user's gaze information, a display control unit 203 that displays on the display unit 210 a plurality of operators associated with the control commands of the musical sound generating device, and displays on the display unit 210 a pointer P for operating the operators based on the gaze information acquired by the acquisition unit 201, a determination unit 204 that determines which of the plurality of operators overlaps with at least a portion of the pointer P, and a communication unit 212 that transmits information conforming to the MIDI standard corresponding to the operator selected by the determination unit 204 to the musical sound generating device 3, thereby enabling a user with limited physical movement to easily control the musical sound generating device 3.

[0076] Any part or all of the functional units described in this specification may be realized by a program. The program mentioned in this specification may be distributed by being non-temporarily recorded on a computer-readable recording medium, or may be distributed via a communication line (including wireless communication) such as the Internet, or may be distributed in a state where it is installed on any terminal.

[0077] Based on the above description, a person skilled in the art may be able to conceive additional effects and various modifications of the present invention, but the aspects of the present invention are not limited to the individual embodiments described above. Various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and spirit of the present invention, which is derived from the content defined in the claims and their equivalents.

[0078] For example, what is described herein as a single device (or component, the same applies hereinafter) (including what is depicted as a single device in the drawings) may be realized by multiple devices. Conversely, what is described herein as multiple devices (including what is depicted as multiple devices in the drawings) may be realized by a single device. Alternatively, some or all of the means and functions included in a certain device (e.g., a server) may be included in another device (e.g., a user terminal).

[0079] Furthermore, not all of the features described in this specification are essential requirements. In particular, features described in this specification but not included in the claims can be considered optional additional features.

[0080] It should be noted that the applicant is merely aware of the inventions disclosed in the documents listed in the "Prior Art Documents" section of this specification, and the present invention does not necessarily aim to solve the problems of the disclosed inventions. The problem that the present invention aims to solve should be determined by taking into consideration the entire specification. For example, if this specification states that a specific configuration achieves a certain effect, it can also be said that the present invention solves a problem that is the reverse of that effect. However, it is not necessarily intended that such a specific configuration be an essential requirement. [Explanation of symbols]

[0081] 1. Musical tone generation system 2 Controller 21 Eye tracking device 22 Input Devices 201 Acquisition Department 202 Reception 203 Display control unit 204 Decision Section 210 Display section 211 Storage section 212 Communications Department 3. Musical tone generator

Claims

1. A controller for controlling a musical sound generating device, an acquisition unit that acquires user's line of sight information; a display control unit that displays a plurality of operators associated with control commands of the musical sound generating device on a display unit, and displays a pointer on the display unit for selecting the operator based on the line-of-sight information acquired by the acquisition unit, and that displays the beat and bar number of the music piece being played near the pointer; a selection unit that selects an operator that overlaps at least a portion of the pointer from among the plurality of operators; a communication unit that transmits information conforming to the MIDI standard corresponding to the operator selected by the determination unit to the musical sound generating device; A controller comprising:

2. a receiving unit that receives an input operation by a user using an input device; The controller according to claim 1 , wherein the determination unit determines the selection of the operator in accordance with an input operation received by the reception unit.

3. the input operation is pressing a switch with a part of the user's body; The controller according to claim 2 , wherein the determination unit determines the selection of the operator based on the user's line of sight information and without any user operation other than pressing the switch.

4. The controller according to claim 1 , wherein the determination unit determines the selection of an operator that has been overlapped with at least a part of the pointer for a predetermined period of time.

5. The controller according to claim 1 , wherein the determination unit determines the selection of an operator that overlaps with the reference position of the pointer.

6. 6. The controller according to claim 5, wherein the reference position is approximately the center of the pointer.

7. The controller according to claim 1 , wherein the display control unit enlarges an area including the pointer and displays the area on the display unit.

8. The controller according to claim 1 , wherein the display control unit displays more of the plurality of operators near the center of the screen of the display unit than at the periphery of the screen.

9. 2. The controller according to claim 1, wherein the plurality of operators include an XY pad capable of simultaneously changing two types of parameters.

10. The controller according to claim 1 , wherein the determination unit determines the selection of the operator without any user operation other than the user's line of sight information.

11. A musical sound generating system comprising a musical sound generating device and a controller for controlling the device, an acquisition unit that acquires user's line of sight information; a display control unit that displays a plurality of operators associated with control commands of the musical sound generating device on a display unit, and displays a pointer on the display unit for selecting the operator based on the line-of-sight information acquired by the acquisition unit, and that displays the beat and bar number of the music piece being played near the pointer; a selection unit that selects an operator that overlaps at least a portion of the pointer from among the plurality of operators; a communication unit that transmits information conforming to the MIDI standard corresponding to the operator selected by the determination unit to the musical sound generating device; A musical sound generating system comprising:

12. 1. A method executed by a controller that controls a musical tone generating device, comprising: acquiring user's gaze information; a step of displaying a plurality of operators associated with control commands of the musical sound generating device on a display unit, and displaying a pointer on the display unit for selecting the operators based on the line-of-sight information, wherein the beat and bar number of the music being played are displayed near the pointer; determining a selection of a control from the plurality of controls that overlaps at least a portion of the pointer; transmitting information conforming to the MIDI standard corresponding to the determined operator to the musical tone generating device; A method comprising:

13. A program for causing a computer to execute a method for controlling a musical sound generating device, the method comprising: a step of displaying a plurality of operators associated with control commands of the musical sound generating device on a display unit, and displaying a pointer on the display unit for selecting the operators based on information of the user's line of sight, the pointer displaying the beat and measure number of the music piece being played near the pointer; determining a selection of a control from the plurality of controls that overlaps at least a portion of the pointer; transmitting information conforming to the MIDI standard corresponding to the determined operator to the musical tone generating device; A program comprising:

14. A computer-readable storage medium storing the program according to claim 13.

Images