Controller, musical sound generation system, method, program, and storage medium

Abstract

To provide a controller which allows a user with limited physical movement to easily control a musical sound generation device, and to provide a musical sound generation system, method, program, and storage medium.SOLUTION: In a musical sound generation system 1, a controller for controlling a musical sound generation device comprises: an acquisition unit for acquiring user line-of-sight information; a display control unit configured to dynamically display, on the display unit, multiple operators associated with control commands of the musical sound generation device, and to display, on the display unit, a pointer for selecting an operator according to the line-of-sight information acquired by the acquisition unit; a determination unit configured to determine selection of an operator overlapping with at least a portion of the pointer from among the multiple operators; and a communication unit configured to send, to the musical sound generation device, information compliant with the MIDI standard and corresponding to the operator whose selection was determined by the determination unit.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a controller, a musical sound generation system, a method, a program, and a storage medium. [Background technology]

[0002] BACKGROUND ART Conventionally, a performance interface is known that generates performance control information for controlling musical tones generated from a musical tone generating device in accordance with the physical condition of an operator (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2001-195059 Summary of the Invention [Problem to be solved by the invention]

[0004] However, nothing has been proposed so far regarding use by users with limited physical movement, such as ALS patients.

[0005] The problem to be solved by the present invention is to enable a user with limited physical movement to easily control a musical sound generating device. [Means for solving the problem]

[0006] The controller of the first embodiment is a controller that controls a musical sound generating device, and includes an acquisition unit that acquires a user's gaze information, a display control unit that dynamically displays a plurality of operators associated with control commands of the musical sound generating device on a display unit and displays a pointer on the display unit for selecting the operator based on the gaze information acquired by the acquisition unit, a determination unit that determines the selection of an operator from the plurality of operators that overlaps with at least a portion of the pointer, and a communication unit that transmits information compliant with the MIDI standard corresponding to the operator that the determination unit has determined to select to the musical sound generating device.

[0007] This configuration allows a user with limited physical movement to easily control the musical sound generating device.

[0008] A controller according to a second aspect is the controller according to the first aspect, wherein the display control unit displays an image of a three-dimensional space on the display unit and displays the plurality of operators on the display unit so that they move in the three-dimensional space.

[0009] According to this configuration, the user can control the musical sound generating device in accordance with the movement of the operator moving in three-dimensional space.

[0010] A controller according to a third aspect is the controller according to the second aspect, wherein the display control unit displays at least one of the plurality of operators on the display unit so that the operator moves from the back to the front in the three-dimensional space.

[0011] According to this configuration, the user can control the musical sound generating device while feeling immersed in the movement of the operator that moves from the back to the front in the three-dimensional space.

[0012] A controller according to a fourth aspect is a controller according to any one of the first to third aspects, wherein the display control unit displays an indicator near the pointer indicating the passage of a predetermined time corresponding to one measure of a piece of music played by the musical sound generating device.

[0013] According to this configuration, the user can control the musical tone generating device in accordance with the passage of the predetermined time indicated by the indicator.

[0014] A controller according to a fifth aspect is the controller according to the fourth aspect, wherein the indicator includes a circular meter graph that dynamically changes as the predetermined time elapses.

[0015] This configuration allows the user to easily visually recognize the passage of a predetermined time.

[0016] A controller according to a sixth aspect is the controller according to the fourth or fifth aspect, wherein the indicator includes a timer that counts down or up the predetermined time.

[0017] According to this configuration, the user can easily control the musical tone generating device in time with the passage of a predetermined time.

[0018] A controller according to a seventh aspect is the controller according to the sixth aspect, wherein the timer counts up or down the predetermined time in beat-by-beat increments.

[0019] With this configuration, the user can control the musical tone generating device in time with the rhythm of the music.

[0020] A controller according to an eighth aspect is the controller according to any one of the fourth to seventh aspects, wherein the decision section decides to select an operator that overlaps at least a part of the pointer when the predetermined time has elapsed.

[0021] According to this configuration, the selection of the operator is automatically determined after a predetermined time has elapsed, making it easy to control the musical tone generating device in accordance with the rhythm of the music.

[0022] A controller according to a ninth aspect is the controller according to any one of the first to eighth aspects, wherein the decision section decides to select an operator that overlaps with a reference position of the pointer.

[0023] This configuration allows for more accurate selection of the controls.

[0024] A controller according to a tenth aspect is the controller according to the ninth aspect, wherein the reference position is approximately the center of the pointer.

[0025] This configuration allows for easier selection of the controls.

[0026] A controller according to an eleventh aspect is the controller according to any one of the first to tenth aspects, wherein the decision section decides the selection of the operator without any user operation other than the user's line of sight information.

[0027] According to this configuration, since an operator can be selected only by the user's line of sight information (line of sight input), even a user with limited physical freedom can easily control the musical sound generating device.

[0028] A musical sound generating system according to a twelfth aspect is a musical sound generating system comprising a musical sound generating device and a controller for controlling the device, and includes an acquisition unit that acquires a user's gaze information, a display control unit that dynamically displays a plurality of operators associated with control commands of the musical sound generating device on a display unit and displays a pointer on the display unit for selecting the operator based on the gaze information acquired by the acquisition unit, a determination unit that determines which of the plurality of operators overlaps with at least a portion of the pointer, and a communication unit that transmits information conforming to the MIDI standard corresponding to the operator whose selection has been determined by the determination unit to the musical sound generating device.

[0029] A method according to a thirteenth aspect is a method for controlling a musical sound generating device, characterized in that it comprises the steps of acquiring a user's gaze information, dynamically displaying a plurality of operators associated with control commands for the musical sound generating device on a display unit, and displaying a pointer on the display unit for selecting the operator based on the gaze information, determining a selection of an operator from the plurality of operators that overlaps with at least a portion of the pointer, and transmitting information conforming to the MIDI standard corresponding to the determined operator to the musical sound generating device.

[0030] This configuration allows a user with limited physical movement to easily control the musical sound generating device.

[0031] A program according to a fourteenth aspect is a program for causing a computer to execute a method for controlling a musical sound generating device, the method comprising the steps of dynamically displaying a plurality of operators associated with control commands for the musical sound generating device on a display unit, and displaying a pointer on the display unit for selecting the operator based on a user's line of sight information; determining a selection of an operator from the plurality of operators that overlaps with at least a portion of the pointer; and transmitting information conforming to the MIDI standard corresponding to the determined operator to the musical sound generating device.

[0032] A storage medium according to a fifteenth aspect is a computer-readable storage medium that stores the program according to the fourteenth aspect. [Effects of the Invention]

[0033] According to the present invention, a user with limited physical movement can easily control a musical sound generating device. [Brief explanation of the drawings]

[0034] [Figure 1] 1 is a diagram showing an example of a system configuration of a musical sound generating system 1 according to an embodiment of the present invention. [Figure 2]1 is a block diagram showing an example of a hardware configuration of a musical sound generating system 1 according to the present embodiment. [Figure 3] 1 is a block diagram showing an example of a functional configuration of a musical sound generating system 1 according to the present embodiment. [Figure 4] 4 is a flowchart showing an example of the operation of the musical sound generating system 1 according to the present embodiment. [Figure 5A] 3A to 3C are diagrams showing an example of a dynamically changing user interface display on the controller 2 according to the present embodiment. [Figure 5B] 5B is a diagram showing an example of a dynamically changing user interface display on the controller 2 according to the present embodiment, showing an example of a display that has changed after FIG. 5A. FIG. [Figure 5C] 5B is a diagram showing an example of a dynamically changing user interface display on the controller 2 according to the present embodiment, showing an example of a display that has changed after FIG. 5B. FIG. [Figure 5D] 5C shows an example of a dynamically changing user interface display on the controller 2 according to the present embodiment, and shows an example of a display that has changed after FIG. 5C. FIG. [Figure 6] 3A to 3C are diagrams showing an example of a plurality of operators displayed on a user interface of the controller 2 according to the present embodiment. [Figure 7] FIG. 5B is a close-up view of the pointer of the user interface shown in FIG. 5A. DETAILED DESCRIPTION OF THE INVENTION

[0035] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In each drawing, components having equivalent functions are designated by the same reference numerals, and detailed description of the components having the same reference numerals will not be repeated.

[0036] (Configuration of musical tone generation system) Fig. 1 is a diagram showing a schematic configuration of a musical sound generation system 1 according to an embodiment. As shown in Fig. 1, the musical sound generation system 1 according to this embodiment includes a controller 2 and a musical sound generation device 3. The controller 2 and the musical sound generation device 3 are communicably connected via a communication interface (I / F) described below.

[0037] The controller 2 is a device for controlling the musical sound generating device 3 based on a communication protocol conforming to the MIDI standard. The controller 2 is connected to an eye-tracking device 21 that allows a user with limited physical movement, such as an ALS patient, to input various information to the controller 2 by eye-tracking, and an input device 22 that allows the user to perform manual operations (the controller 2 may be equipped with the eye-tracking device 21 and the input device 22).

[0038] The eye tracking device 21 tracks the movement of the user's eyes (gaze direction) and outputs gaze information to the controller 2. The gaze information can be generated, for example, by extracting characteristic patterns such as pupils from images of the user's eyes captured sequentially. This embodiment focuses on the fact that ALS patients can move their gaze, and controls the musical sound generating device 3 using the gaze information.

[0039] The input device 22 is an input device that can be operated by a user, such as a manual switch or a foot switch. When a user performs an operation using the input device 22, a signal corresponding to the operation is output from the input device 22 to the controller 2. The input device 22 is preferably one that allows a user with limited physical movement, such as an ALS patient, to easily perform input operations, and the input operation is, for example, to press a switch with the user's body.

[0040] The musical sound generating device 3 is a device for generating musical sounds based on signals conforming to the MIDI standard received from the controller 2. For example, a personal computer can be used as a musical sound generating device by installing and running integrated music software called a DAW (Digital Audio Workstation) on the personal computer. The musical sound generating device 3 also includes a sound system 31 for emitting sounds based on acoustic signals.

[0041] (Hardware configuration) Next, a description will be given of the hardware configuration of the musical sound generating system 1 according to this embodiment. Fig. 2 is a block diagram showing an example of the hardware configuration of the musical sound generating system 1 according to this embodiment.

[0042] In the controller 2, the CPU 23 is a processing device that controls the overall operation of the controller 2. The ROM 24 is a non-volatile memory that stores the control programs executed by the CPU 23 and various data. The RAM 25 is a volatile memory that is used as a load area and work area for the programs executed by the CPU 23. The display 26 is a display device that displays various information (user interface). The storage device 27 is storage means for storing various information, and may be built into the controller 2 body or may have a removable storage medium. The communication interface (I / F) 28 is an interface for connecting to the musical sound generating device 3. The bus 29 is a bus line that connects the above components to each other.

[0043] In this embodiment, the musical sound generating device 3 is a PC with DAW software installed. By executing the DAW software, various music production functions (for example, hard disk recording function, MIDI data and audio data creation and editing function, mixing function, sequencer function, etc.) can be realized. Note that, although this embodiment will be described using an example in which the musical sound generating device 3 is configured with a PC and DAW software, it may also be a standalone sequencer device.

[0044] In the musical sound generating device 3, the CPU 32, ROM 33, RAM 34, storage device 36, input device 37 such as a mouse and keyboard, and display 38 are basic components of a general-purpose PC. The communication I / F 35 is an interface for connecting to the controller 2. The communication I / Fs 28 and 35 may be, for example, a wired music I / F such as MIDI, a general-purpose wired I / F such as USB or IEEE1394, or a general-purpose wireless I / F such as wireless LAN or Bluetooth (registered trademark). The sound system 31 emits analog audio signals passed from a codec, which is a sound I / O equipped with a digital-to-analog (D / A) conversion function. This sound system 31 is an added function in the form of a board or card inserted into a slot provided in the PC constituting the musical sound generating device 3. The bus 39 is a bus line that interconnects the above components.

[0045] (Functional configuration) Next, a description will be given of the functional configuration of the musical sound generating system 1 according to this embodiment. Fig. 3 is a diagram showing an example of the functional configuration of the musical sound generating system 1 according to this embodiment.

[0046] As shown in FIG. 3, the controller 2 includes a control unit 200, a display unit 210, a storage unit 211, and a communication unit 212.

[0047] The control unit 200 includes an acquisition unit 201 , a reception unit 202 , a determination unit 204 , and a display control unit 203 .

[0048] The acquisition unit 201 acquires gaze information of the user of the controller 2. Specifically, the acquisition unit 201 acquires information (gaze information) relating to the gaze direction of the user output from the eye tracking device 21.

[0049] The receiving unit 202 receives an input operation by a user using the input device 22 (a manual switch, a foot switch, etc.).

[0050] The display control unit 203 dynamically displays, on the display unit 210 (corresponding to the display 26 in FIG. 2), a plurality of operators (icons) associated with the control commands of the musical sound generating device 3, together with various information. Furthermore, the display control unit 203 displays, on the display unit 210, a pointer for operating the operators, based on the user's line of sight information acquired by the acquisition unit 201.

[0051] 5A to 5D are diagrams showing an example of a user interface 50 that is dynamically displayed on the display unit 210 by the display control unit 203. The display control unit 203 changes the display of the user interface 50 in the order of Fig. 5A, Fig. 5B, Fig. 5C, and Fig. 5D over time.

[0052] As shown in FIGS. 5A to 5D, the display control unit 203 displays an image of a three-dimensional space 51 on the display unit 210 as the user interface 50. The display control unit 203 then displays a plurality of controls 60 that move from the back to the front of the three-dimensional space 51. The multiple controls 60 disappear when they reach the frontmost position in the three-dimensional space, but in parallel with this, new multiple controls 60 are generated on the screen and displayed as moving from the back to the front. Displaying the user interface 50 in this manner gives the user the sensation that the multiple controls 60 are approaching the user from the back of the screen of the display unit 210, or that the user is moving in the depth direction through the three-dimensional space 51. This allows the user to operate the controller 2 with a sense of immersion.

[0053] Fig. 6 is a diagram showing an example of a plurality of operators 60 displayed on the user interface 50. As shown in Fig. 6, the plurality of operators 60 include nine rectangular operators, 61 to 69. Each of these operators 61 to 69 is associated with a control command of the musical sound generation device 3 (DAW), and when the user operates (selects) each of the operators 61 to 69, the musical sound generation device 3 performs processing according to the control command.

[0054] 5A to 5D, the user interface 50 includes a pointer P for operating / selecting each of the above-mentioned controls. The pointer P is displayed on the display unit 210 based on the line-of-sight information acquired by the acquisition unit 201. Specifically, the position of the point of gaze on the screen of the display unit 210 is obtained from the line-of-sight information, and the pointer P is displayed at that position.

[0055] FIG. 7 is an enlarged view of the pointer P displayed on the user interface 50 shown in FIGS. 5A to 5D. As shown in FIG. 7, the pointer P includes a horizontal reference line Px and a vertical reference line Py, and the intersection of the reference lines Px and Py is the reference position. This reference position is located at the position of the user's gaze point obtained from the gaze information. The pointer P also includes a rectangular area P1 and an indicator P2 displayed within this rectangular area. The rectangular area P1 is an area where the determination unit 204, described below, determines whether the pointer P overlaps with a control. The indicator P2 is an indicator that indicates the passage of a predetermined time corresponding to one measure of a musical piece played by the musical sound generating device 3. The indicator P2 includes a circular meter graph P21 that dynamically changes as the predetermined time passes and a timer P22 that counts down the predetermined time. For example, in the case of a musical piece in 4 / 4 time, one measure is made up of four beats, so the timer P22 counts down one beat at a time in the order of 4, 3, 2, and 1. Note that the timer P22 may count up instead of counting down, and the count down or count up does not have to be in beat increments. For example, the timer P22 may count up or down in one-second increments. In this way, the pointer P has an indicator P2 that indicates the passage of a predetermined time corresponding to one measure of the music piece played by the musical sound generating device 3, so that the user of the controller 2 can easily control the musical sound generating device 3 in time with the rhythm of the music piece.

[0056] Returning to FIG. 3 , the determination unit 204 determines the selection of an operator from among the plurality of operators that overlaps with at least a part of the rectangular area P1 of the pointer P at the timing when the timer counts up (when a predetermined time has elapsed). This allows the selection of an operator to be determined in time with the rhythm of the music solely through gaze input using the eye tracking device 21. The determination unit 204 may also be configured to determine the selection of an operator that overlaps with the reference position of the pointer P. This reduces the risk of erroneous operation (erroneous selection) of an operator when the operators are arranged adjacent to each other. This selection method, which allows the selection of an operator to be performed solely through gaze information, is also useful for users who have difficulty performing actions other than moving their gaze.

[0057] Furthermore, the determination unit 204 may determine the selection of an operator in accordance with an input operation by the user using the input device 22, which is received by the receiving unit 202. That is, the user can select an operator by gaze input using the eye tracking device 21, and then determine the selection of the operator using the input device 22 (such as a manual switch or a foot switch). In this way, the user can operate the controller 2 while experiencing the rhythm of the music played by the musical sound generating device 3. This selection method is useful for a user who can operate the input device 22 with a part of their body, such as a hand or a foot, in addition to moving their gaze.

[0058] The storage unit 211 stores information relating to each control command of the musical sound generating device 3 (DAW) associated with each operator.

[0059] The communication unit 212 transmits to the musical sound generating device 3 information (control command information, MIDI event information) conforming to the MIDI standard corresponding to the operator selected by the determination unit 204 .

[0060] Next, we will explain the functional configuration of the musical sound generating device 3. As shown in Fig. 3, the musical sound generating device 3 has an audio input unit 301, a sequencer 302, a recorder 303, a sound source 304, a mixer 305, a sound system 306, and a communication unit 307.

[0061] The audio input unit 301 receives input of an analog audio signal converted into a digital form, or inputs a digital audio signal itself.

[0062] The sequencer 302 records the MIDI event information received by the communication unit 307. This MIDI event information is supplied to the sound source 304, which will be described later.

[0063] The sound source 304 is a software sound source. The sound source 304 generates an audio signal in response to input MIDI event information. The audio signal output from the sound source 304 is input to the recorder 303 and the mixer 305.

[0064] The recorder 303 records the audio signal input by the audio input unit 301 and the audio signal output from the sound source 304. The audio signal recorded in the recorder 303 is output to the mixer 305 in response to a playback instruction.

[0065] The mixer 305 supplies the audio signals output from the sound source 304 and the recorder 303 to the sound system 306 .

[0066] (Example of operation) Next, a description will be given of an example of the operation of the musical sound generating system 1. Fig. 4 is a flowchart showing an example of the operation of the musical sound generating system 1.

[0067] First, the acquisition unit 201 of the control unit 200 in the controller 2 acquires the user's gaze information output from the eye tracking device 21 (step S1). The gaze information may be acquired at any timing, for example, constantly in real time.

[0068] Next, the display control section 203 of the control section 200 dynamically displays on the display section 210 a plurality of operators associated with the control commands of the musical sound generating device 3 (DAW) (step S2).

[0069] Furthermore, the display control unit 203 of the control unit 200 displays a pointer for operating the operator on the display unit 210 based on the line-of-sight information acquired by the acquisition unit 201 (step S3).

[0070] Next, the determination unit 204 of the control unit 200 determines the selection of an operator that overlaps at least a part of the pointer from among the multiple operators (step S4).

[0071] Next, the communication unit 307 transmits information conforming to the MIDI standard corresponding to the operator selected by the determination unit 204 to the musical sound generating device 3 (step S5).

[0072] Then, the musical sound generating device 3 performs various processes (execution of control commands, recording of MIDI events, etc.) based on the information conforming to the MIDI standard received from the controller 2 (step S6).

[0073] As described above, according to this embodiment, the musical sound generating system 1 includes an acquisition unit 201 that acquires the user's gaze information, a display control unit 203 that dynamically displays on the display unit 210 a plurality of operators associated with the control commands of the musical sound generating device, and displays on the display unit 210 a pointer P for operating the operators based on the gaze information acquired by the acquisition unit 201, a determination unit 204 that determines which of the plurality of operators overlaps with at least a portion of the pointer P, and a communication unit 212 that transmits information conforming to the MIDI standard corresponding to the operator selected by the determination unit 204 to the musical sound generating device 3, thereby enabling a user with limited physical movement to easily control the musical sound generating device 3. Any part or all of the functional units described in this specification may be realized by a program. The program mentioned in this specification may be distributed by being non-temporarily recorded on a computer-readable recording medium, or may be distributed via a communication line (including wireless communication) such as the Internet, or may be distributed in a state where it is installed on any terminal.

[0074] Based on the above description, a person skilled in the art may be able to conceive additional effects and various modifications of the present invention, but the aspects of the present invention are not limited to the individual embodiments described above. Various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and spirit of the present invention, which is derived from the content defined in the claims and their equivalents.

[0075] For example, what is described herein as a single device (or component, the same applies hereinafter) (including what is depicted as a single device in the drawings) may be realized by multiple devices. Conversely, what is described herein as multiple devices (including what is depicted as multiple devices in the drawings) may be realized by a single device. Alternatively, some or all of the means and functions included in a certain device (e.g., a server) may be included in another device (e.g., a user terminal).

[0076] Furthermore, not all of the features described in this specification are essential requirements. In particular, features described in this specification but not included in the claims can be considered optional additional features.

[0077] It should be noted that the applicant is merely aware of the inventions disclosed in the documents listed in the "Prior Art Documents" section of this specification, and the present invention does not necessarily aim to solve the problems of the disclosed inventions. The problem that the present invention aims to solve should be determined by taking into consideration the entire specification. For example, if this specification states that a specific configuration achieves a certain effect, it can also be said that the present invention solves a problem that is the reverse of that effect. However, it is not necessarily intended that such a specific configuration be an essential requirement. [Explanation of symbols]

[0078] 1. Musical tone generation system 2 Controller 21 Eye tracking device 22 Input Devices 201 Acquisition Department 202 Reception 203 Display control unit 204 Decision Section 210 Display section 211 Storage section 212 Communications Department 3. Musical tone generator

Claims

1. A controller for controlling a musical sound generating device, an acquisition unit that acquires user's line of sight information; a display control unit that dynamically displays, on a display unit, a plurality of operators associated with control commands of the musical sound generating device, so as to move integrally from the back to the front in a three-dimensional space, and that displays, on the display unit, a pointer for selecting the operator based on the line-of-sight information acquired by the acquisition unit; a selection unit that selects an operator that overlaps at least a portion of the pointer from among the plurality of operators; a communication unit that transmits information conforming to the MIDI standard corresponding to the operator selected by the determination unit to the musical sound generating device; A controller comprising:

2. The controller according to claim 1 , wherein the display control unit displays an image of a three-dimensional space on the display unit, and displays the plurality of operators on the display unit so as to move in the three-dimensional space.

3. The controller according to claim 2 , wherein the display control unit displays at least one of the plurality of operators on the display unit so as to move from the back to the front in the three-dimensional space.

4. 2. The controller according to claim 1, wherein the display control unit displays an indicator near the pointer that indicates the passage of a predetermined time corresponding to one measure of a piece of music played by the musical sound generating device.

5. 5. The controller of claim 4, wherein the indicator includes a circular meter graph that dynamically changes as the predetermined time elapses.

6. 5. The controller of claim 4, wherein the indicator includes a timer that counts down or up the predetermined time.

7. 7. The controller according to claim 6, wherein the timer counts up or down the predetermined time in beat increments.

8. 8. The controller according to claim 4, wherein the determination unit determines the selection of an operator that overlaps at least a part of the pointer after the predetermined time has elapsed.

9. The controller according to claim 1 , wherein the determination unit determines the selection of an operator that overlaps with the reference position of the pointer.

10. The controller according to claim 9 , wherein the reference position is approximately the center of the pointer.

11. The controller according to claim 1 , wherein the determination unit determines the selection of the operator without any user operation other than the user's line of sight information.

12. A musical sound generating system comprising a musical sound generating device and a controller for controlling the device, an acquisition unit that acquires user's line of sight information; A plurality of operators associated with control commands of a musical sound generating device are dynamically displayed integrally on a display unit so as to move from the back to the front in a three-dimensional space, and a display unit is provided for selecting the operators based on the line-of-sight information acquired by the acquisition unit. a display control unit that displays a pointer for the a selection unit that selects an operator that overlaps at least a portion of the pointer from among the plurality of operators; a communication unit that transmits information conforming to the MIDI standard corresponding to the operator selected by the determination unit to the musical sound generating device; A musical sound generating system comprising:

13. 1. A method executed by a controller that controls a musical tone generating device, comprising: acquiring user's gaze information; a step of dynamically displaying a plurality of operators associated with control commands of the musical sound generating device on a display unit so as to move from the back to the front in a three-dimensional space, and displaying a pointer on the display unit for selecting the operators based on the line-of-sight information; determining a selection of a control from the plurality of controls that overlaps at least a portion of the pointer; transmitting information conforming to the MIDI standard corresponding to the determined operator to the musical tone generating device; A method comprising:

14. A program for causing a computer to execute a method for controlling a musical sound generating device, the method comprising: a step of dynamically displaying a plurality of operators associated with control commands of the musical sound generating device on a display unit so as to move from the back to the front in a three-dimensional space, and displaying a pointer on the display unit for selecting the operators based on user line-of-sight information; determining a selection of a control from the plurality of controls that overlaps at least a portion of the pointer; transmitting information conforming to the MIDI standard corresponding to the determined operator to the musical tone generating device; A program comprising:

15. A computer-readable storage medium storing the program according to claim 14.

Images