Controller, musical sound generation system, method, program, and storage medium

Abstract

To provide a controller which enables a user with limited body movement to easily control a musical sound generation device, a musical sound generation system, a method, a program, and a storage medium.SOLUTION: In a musical sound generation system 1, the controller for controlling the musical sound generation device comprises: an acquisition part for acquiring user's line-of-sight information; a display control part that displays on a display part a plurality of operators associated with control commands of the musical sound generation device and displays on the display part a pointer for selecting an operator based on the line-of-sight information acquired by the acquisition part; a determination part that determines a selection of an operator overlapping at least a portion of the pointer from among the plurality of operators; and a communication part that transmits, to the musical sound generation device, information compliant with MIDI standards corresponding to the operator selected upon determination by the determination part. The plurality of operators include at least one fader operator that switches a sound source being played from a first sound source to a second sound source.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a controller, a musical sound generation system, a method, a program, and a storage medium. [Background technology]

[0002] BACKGROUND ART Conventionally, a performance interface is known that generates performance control information for controlling musical tones generated from a musical tone generating device in accordance with the physical condition of an operator (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2001-195059 Summary of the Invention [Problem to be solved by the invention]

[0004] However, nothing has been proposed so far regarding use by users with limited physical movement, such as ALS patients.

[0005] The problem to be solved by the present invention is to enable a user with limited physical movement to easily control a musical sound generating device. [Means for solving the problem]

[0006] [1] A controller for controlling a musical sound generating device, an acquisition unit that acquires user's line of sight information; a display control unit that displays a plurality of operators associated with the control commands of the musical sound generating device on a display unit, and displays a pointer on the display unit for selecting the operators based on the line-of-sight information acquired by the acquisition unit; a selection unit that selects an operator that overlaps at least a part of the pointer from among the plurality of operators; a communication unit that transmits information conforming to the MIDI standard corresponding to the operator selected by the determination unit to the musical sound generating device, The controller, wherein the plurality of operators include at least one fader operator for switching the sound source being reproduced from a first sound source to a second sound source.

[0007] [2] The controller described in [1] above, wherein the at least one fader operator includes a crossfader operator including a shaft and a knob, and the crossfader operator continuously changes the volume of each of the first sound source and the second sound source depending on the position of the knob on the shaft.

[0008] [3] The controller described in [1] or [2] above, wherein the at least one fader operator includes a fader pattern operator for continuously changing the volume of each of the first sound source and the second sound source according to a pre-stored pattern.

[0009] [4] The controller described in any one of [1] to [3] above, wherein the plurality of operators include a first group of operators that perform operations on the first sound source and a second group of operators that perform operations on the second sound source.

[0010] [5] The controller described in [4] above, wherein the display control unit highlights the first group of operators when the volume of the first sound source is louder than the volume of the second sound source, and highlights the second group of operators when the volume of the second sound source is louder than the volume of the first sound source.

[0011] [6] The controller described in [4] or [5] above, wherein the determination unit determines the selection of an operator included in the first operator group when the volume of the first sound source is louder than the volume of the second sound source, and determines the selection of an operator included in the second operator group when the volume of the second sound source is louder than the volume of the first sound source.

[0012] [7] the first operator group includes a sound source selection operator for selecting an odd-numbered sound source from among a plurality of sound sources to be played back consecutively; The controller according to any one of [4] to [6] above, wherein the first operator group includes a sound source selection operator for selecting an even-numbered sound source from among a plurality of sound sources to be played back consecutively.

[0013] [8] the first operator group is displayed on the left side of the display unit, The controller according to [7] above, wherein the second group of operators is displayed on the right side of the display unit.

[0014] [9] The controller described in [8] above, wherein the plurality of operators include a plurality of clip selection operators for selecting a video clip to display while a sound source is being played, and the plurality of clip selection operators are displayed between the first group of operators and the second group of operators.

[0015]

[10] The controller according to any one of [4] to [9] above, wherein the first operator group and the second operator group include an effect operator for applying an effect to a sound source being played back.

[0016]

[11] The controller according to any one of [4] to

[10] above, wherein the first group of operators and the second group of operators include an XY pad capable of changing two types of parameters simultaneously.

[0017]

[12] The controller according to any one of [1] to

[11] above, wherein the plurality of operators includes a screen switching operator for switching to a screen on which other operators are displayed.

[0018]

[13] a receiving unit that receives an input operation by a user using an input device; The controller according to any one of [1] to

[12] above, wherein the determination unit determines the selection of the operator in accordance with the input operation received by the reception unit.

[0019]

[14] the input operation is pressing a switch with a part of the user's body; The controller according to

[13] above, wherein the determination unit determines the selection of the operator based on the user's line of sight information and without any user operation other than pressing the switch.

[0020]

[15] The controller according to any one of [1] to

[14] above, wherein the determination unit determines the selection of an operator that has been overlapping with at least a part of the pointer for a predetermined period of time.

[0021]

[16] The controller according to any one of [1] to

[15] above, wherein the determination unit determines the selection of an operator that overlaps with a reference position of the pointer.

[0022]

[17] The controller according to

[16] above, wherein the reference position is approximately the center of the pointer.

[0023]

[18] The controller according to any one of [1] to

[17] above, wherein the display control unit displays the beat and bar number of the music being played near the pointer.

[0024]

[19] The controller according to any one of [1] to

[18] above, wherein the determination unit determines the selection of the operator without any user operation other than the user's line of sight information.

[0025]

[20] A musical sound generating system comprising a musical sound generating device and a controller for controlling the device, an acquisition unit that acquires user's line of sight information; a display control unit that displays a plurality of operators associated with the control commands of the musical sound generating device on a display unit, and displays a pointer on the display unit for selecting the operators based on the line-of-sight information acquired by the acquisition unit; a selection unit that selects an operator that overlaps at least a part of the pointer from among the plurality of operators; a communication unit that transmits information conforming to the MIDI standard corresponding to the operator selected by the determination unit to the musical sound generating device, The musical sound generating system, wherein the plurality of operators include at least one fader operator for switching the sound source being reproduced from a first sound source to a second sound source.

[0026] [twenty one] 1. A method for controlling a musical tone generating device, comprising: acquiring user's gaze information; displaying a plurality of operators associated with control commands of the musical sound generating device on a display unit, and displaying a pointer on the display unit for selecting the operators based on the line-of-sight information; determining a selection of a control from the plurality of controls that overlaps at least a portion of the pointer; and transmitting information conforming to the MIDI standard corresponding to the determined operator to the musical sound generating device, The method, wherein the plurality of controls includes at least one fader control for switching the currently played sound source from a first sound source to a second sound source.

[0027] [twenty two] A program for causing a computer to execute a method for controlling a musical sound generating device, the method comprising: a step of displaying a plurality of operators associated with control commands of the musical sound generating device on a display unit, and displaying a pointer on the display unit for selecting the operators based on user line-of-sight information; determining a selection of a control from the plurality of controls that overlaps at least a portion of the pointer; and transmitting information conforming to the MIDI standard corresponding to the determined operator to the musical sound generating device, The program, wherein the plurality of operators include at least one fader operator for switching the sound source being played from a first sound source to a second sound source.

[0028] [twenty three] A computer-readable storage medium storing the program described in

[22] above. [Effects of the Invention]

[0029] According to the present invention, a user with limited physical movement can easily control a musical sound generating device. [Brief explanation of the drawings]

[0030] [Figure 1] 1 is a diagram showing an example of a system configuration of a musical sound generating system 1 according to an embodiment of the present invention. [Figure 2] 1 is a block diagram showing an example of a hardware configuration of a musical sound generating system 1 according to the present embodiment. [Figure 3] 1 is a block diagram showing an example of a functional configuration of a musical sound generating system 1 according to the present embodiment. [Figure 4] 4 is a flowchart showing an example of the operation of the musical sound generating system 1 according to the present embodiment. [Figure 5] FIG. 2 is a diagram showing an example of a user interface of the controller 2 according to the present embodiment. [Figure 6] FIG. 6 is an enlarged view of the pointer P shown in FIG. 5. [Figure 7] 3A and 3B are diagrams showing examples of fader curves associated with fader operators according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0031] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In each drawing, components having equivalent functions are designated by the same reference numerals, and detailed description of the components having the same reference numerals will not be repeated.

[0032] (Configuration of musical tone generation system) Fig. 1 is a diagram showing a schematic configuration of a musical sound generation system 1 according to an embodiment. As shown in Fig. 1, the musical sound generation system 1 according to this embodiment includes a controller 2 and a musical sound generation device 3. The controller 2 and the musical sound generation device 3 are communicably connected via a communication interface (I / F) described below.

[0033] The controller 2 is a device for controlling the musical sound generating device 3 based on a communication protocol conforming to the MIDI standard. The controller 2 is connected to an eye-tracking device 21 that allows a user with limited physical movement, such as an ALS patient, to input various information to the controller 2 by eye-tracking, and an input device 22 that allows the user to perform manual operations (the controller 2 may be equipped with the eye-tracking device 21 and the input device 22).

[0034] The eye tracking device 21 tracks the movement of the user's eyes (gaze direction) and outputs gaze information to the controller 2. The gaze information can be generated, for example, by extracting characteristic patterns such as pupils from images of the user's eyes captured sequentially. This embodiment focuses on the fact that ALS patients can move their gaze, and controls the musical sound generating device 3 using the gaze information.

[0035] The input device 22 is an input device that can be operated by a user, such as a manual switch or a foot switch. When a user performs an operation using the input device 22, a signal corresponding to the operation is output from the input device 22 to the controller 2. The input device 22 is preferably one that allows a user with limited physical movement, such as an ALS patient, to easily perform input operations, and the input operation is, for example, to press a switch with the user's body.

[0036] The musical sound generating device 3 is a device for generating musical sounds based on signals conforming to the MIDI standard received from the controller 2. For example, a personal computer can be used as a musical sound generating device by installing and running integrated music software called a DAW (Digital Audio Workstation) on the personal computer. The musical sound generating device 3 also includes a sound system 31 for emitting sounds based on acoustic signals.

[0037] (Hardware configuration) Next, a description will be given of the hardware configuration of the musical sound generating system 1 according to this embodiment. Fig. 2 is a block diagram showing an example of the hardware configuration of the musical sound generating system 1 according to this embodiment.

[0038] In the controller 2, the CPU 23 is a processing device that controls the overall operation of the controller 2. The ROM 24 is a non-volatile memory that stores the control programs executed by the CPU 23 and various data. The RAM 25 is a volatile memory that is used as a load area and work area for the programs executed by the CPU 23. The display 26 is a display device that displays various information (user interface). The storage device 27 is storage means for storing various information, and may be built into the controller 2 body or may have a removable storage medium. The communication interface (I / F) 28 is an interface for connecting to the musical sound generating device 3. The bus 29 is a bus line that connects the above components to each other.

[0039] In this embodiment, the musical sound generating device 3 is a PC with DAW software installed. By executing the DAW software, various music production functions (for example, hard disk recording function, MIDI data and audio data creation and editing function, mixing function, sequencer function, etc.) can be realized. Note that, although this embodiment will be described using an example in which the musical sound generating device 3 is configured with a PC and DAW software, it may also be a standalone sequencer device.

[0040] In the musical sound generating device 3, the CPU 32, ROM 33, RAM 34, storage device 36, input device 37 such as a mouse and keyboard, and display 38 are basic components of a general-purpose PC. The communication I / F 35 is an interface for connecting to the controller 2. The communication I / Fs 28 and 35 may be, for example, a wired music I / F such as MIDI, a general-purpose wired I / F such as USB or IEEE1394, or a general-purpose wireless I / F such as wireless LAN or Bluetooth (registered trademark). The sound system 31 emits analog audio signals passed from a codec, which is a sound I / O equipped with a digital-to-analog (D / A) conversion function. This sound system 31 is an added function in the form of a board or card inserted into a slot provided in the PC constituting the musical sound generating device 3. The bus 39 is a bus line that interconnects the above components.

[0041] (Functional configuration) Next, a description will be given of the functional configuration of the musical sound generating system 1 according to this embodiment. Fig. 3 is a diagram showing an example of the functional configuration of the musical sound generating system 1 according to this embodiment.

[0042] As shown in FIG. 3, the controller 2 includes a control unit 200, a display unit 210, a storage unit 211, and a communication unit 212.

[0043] The control unit 200 includes an acquisition unit 201 , a reception unit 202 , a determination unit 204 , and a display control unit 203 .

[0044] The acquisition unit 201 acquires gaze information of the user of the controller 2. Specifically, the acquisition unit 201 acquires information (gaze information) relating to the gaze direction of the user output from the eye tracking device 21.

[0045] The receiving unit 202 receives an input operation by a user using the input device 22 (a manual switch, a foot switch, etc.).

[0046] The display control unit 203 displays, on the display unit 210 (corresponding to the display 26 in FIG. 2), a plurality of operators (icons) associated with the control commands of the musical sound generating device 3 together with various information. In addition, the display control unit 203 displays, on the display unit 210, a pointer P for operating the operators, based on the user's line of sight information acquired by the acquisition unit 201.

[0047] 5 is a diagram showing an example of a user interface 50 displayed on the display unit 210 by the display control unit 203. As shown in Fig. 5, the user interface 50 includes, as a plurality of operators, a first operator group 51 (DECK_A), a second operator group 52 (DECK_B), a crossfader operator (scroll bar) 53 which is a fader operator, a fader pattern operator 54, a clip selection operator 55, a stop operator 56, and a screen switching operator 57. Each of these operators (including operators included in the operator group) is associated with a control command of the musical sound generation device 3 (DAW), and when the user operates (selects) each operator, the musical sound generation device 3 performs processing according to the control command.

[0048] The first operator group 51 (DECK_A) is a group of operators that the user operates for the first sound source (an odd-numbered sound source among multiple sound sources to be played back consecutively), and is displayed on the left side of the screen displaying the user interface 50. The first operator group 51 includes an XY pad 51a, at least one sound source selection operator 51b, and at least one effect operator 51c.

[0049] The XY pad 51a is a control that allows the user to perform two-dimensional input operations and can simultaneously change two types of parameters. Two types of parameters can be simultaneously changed by moving a pointer P on the XY pad 51a. Operation of the XY pad 51a may be enabled while the reception unit 202 is receiving operations from the input device 22. For example, the user may operate the XY pad 51a (change two types of parameters) by moving the pointer P on the XY pad 51a using eye-gaze input while pressing a manual switch (input device 22). The sound source selection operator 51b is a control for selecting a sound source (music) to be played (performed). The first operator group 51 includes multiple sound source selection operators (Song_1, Song_3, Song_5, Song_7, Song_9) for selecting odd-numbered sound sources (first sound sources) from multiple sound sources to be played consecutively. The effect operator 51c is a control for executing audio effects.

[0050] The second operator group 52 (DECK_B) is a group of operators that the user operates on the second sound source (an even-numbered sound source among multiple sound sources to be played back consecutively), and is displayed on the right side of the screen displaying the user interface 50. The second operator group 52 includes an XY pad 52a, at least one sound source selection operator 52b, and at least one effect operator 52c.

[0051] The functions of the XY pad 52a and the effect operator 52c are similar to those of the XY pad 51a and the effect operator 51c described above, and therefore detailed description thereof will be omitted.

[0052] The sound source selection operator 52b is an operator for selecting a sound source (music) to be played (performed). The second operator group 52 includes a plurality of sound source selection operators (Song_2, Song_4, Song_6, Song_8, Song_10) for selecting an even-numbered sound source (second sound source) from among a plurality of sound sources to be played consecutively.

[0053] The crossfader operator 53 is an operator (fader operator) for switching the currently played sound source from the first sound source to the second sound source. The crossfader operator 53 includes a substantially horizontal shaft 53a and a knob 53b that can move the shaft 53a.

[0054] FIG. 7 is a diagram showing an example of the relationship (fader curve) between the horizontal position (position on the axis 53a) of the knob 53b of the crossfader control 53 and the volumes of the first and second sound sources. As shown in FIG. 7, as the knob 53b moves from the left end to the right end on the axis 53a, the volume of the first sound source decreases and the volume of the second sound source increases. In other words, by moving the knob 53b of the crossfader control 53 from the left end to the right end, the sound source being played can be switched from the first sound source to the second sound source. Note that the ratio (V2 / (V1+V2)) of the volume of the second sound source to the total volume (V1+V2[dB]), which is the sum of the volume of the first sound source (V1[dB]) and the volume of the second sound source (V2[dB]), may increase as the knob 53b moves from the left end to the right end on the axis 53a.

[0055] Furthermore, the speed at which the crossfader control 53 switches from the first sound source to the second sound source can be changed by changing the speed at which the knob 53b is moved on the shaft 53a. For example, when the knob 53b is slowly moved from the left end to the right end on the shaft 53a, the sound source can be switched slowly from the first sound source to the second sound source, and when the knob 53b is quickly moved from the left end to the right end on the shaft 53a, the sound source can be switched slowly from the first sound source to the second sound source.

[0056] Immediately after using the crossfader operator 53 to switch the currently playing sound source from a first sound source (for example, "Song_1" shown in FIG. 5) to a second sound source (for example, "Song_2" shown in FIG. 5), the knob 53b of the crossfader operator 53 moves to the far right. In this state, moving the knob 53b of the crossfader operator 53 from the far right to the far left switches the currently playing sound source from the second sound source to the first sound source. At this time, by selecting an arbitrary sound source (for example, "Song_3" shown in FIG. 5) in advance with the sound source selection operator 51b of the first operator group 51, the sound source can be switched seamlessly according to the user's intention (for example, switching between sound sources in the order "Song_1", "Song_2", and "Song_3").

[0057] Here, the reason why the first operator group 51 is used for odd-numbered sound sources (first sound sources) and the second operator group 52 is used for even-numbered sound sources (second sound sources) will be explained. The crossfader operator 53 can switch the currently playing sound source (music) from the first sound source to the second sound source (make the volume of the second sound source louder than the volume of the first sound source) by moving the knob 53b from the left end to the right end on the horizontally displayed axis 53a. For this reason, the first operator group 51 for the first sound source (odd-numbered sound source) is arranged on the left side of the screen and the second operator group 52 for the second sound source (even-numbered sound source) is arranged on the right side of the screen to correspond to the behavior (movement direction) of the knob 53b of the crossfader operator 53. This shortens the distance of line-of-sight movement required to operate the second sound source using the second operator group after switching the sound source using the crossfader operator 53, allowing for smooth operation of the operators. That is, the user can smoothly select operators in the order of first operator group 51, crossfader operator 53, and second operator group 52 from the left side of the screen.

[0058] The fader pattern operator 54 is an operator (fader operator) for switching between sound sources at a pre-stored speed. As described above, the crossfader operator 53 switches the sound source being played from the first sound source to the second sound source by moving the knob 53b on the shaft 53a arranged substantially horizontally, but it takes some getting used to moving the knob 53b on the shaft 53a by inputting with one's line of sight.

[0059] Therefore, in the controller 2 of this embodiment, sound source switching speed patterns preferred by the user (such as a slow switching pattern or a fast switching pattern) are stored in the storage unit 211 or the like, and each stored switching pattern is assigned to each fader pattern operator 54 (three types, "CF_1," "CF_2," and "CF_3," in the example shown in FIG. 5). This allows the user to switch the sound source being played at the desired sound source switching speed by selecting the fader pattern operator 54 to which the desired sound source switching pattern is assigned, without having to perform any elaborate operation of the crossfader operator 53.

[0060] After the sound source being played is switched from the first sound source to the second sound source using the fader pattern operator 54, the sound source being played can be switched from the second sound source to the first sound source by selecting the desired fader pattern operator 54 again.

[0061] Here, a description will be given of display control linked to switching of sound sources using fader operators (crossfader operator 53 and fader pattern operator 54). After switching from the first sound source to the second sound source, the second operator group 52 will be mainly operated, so the second operator group 52 may be highlighted. Conversely, after switching from the second sound source to the first sound source, the first operator group 51 will be mainly operated, so the first operator group 51 may be highlighted. That is, the display control unit 203 may highlight the first operator group when the volume of the first sound source is greater than the volume of the second sound source, and may highlight the second operator group when the volume of the second sound source is greater than the volume of the first sound source.

[0062] Clip selection operators 55 are operators for selecting a video clip to be displayed during playback of an audio source. Video clips stored in storage unit 211 or the like are assigned in advance to clip selection operators 55 (ten operators "FX_1" to "FX_10" in the example shown in FIG. 5), and the user can select any clip selection operator to display the desired video clip on display unit 210 or an external display unit (not shown).

[0063] The stop operator 56 is an operator for stopping the playback of the sound source (music), and is made larger in size than the other operators to make it easier for the user to stop the playback of the sound source at a timing desired by the user.

[0064] The screen switching operator 57 is an operator for switching the screen to a screen (not shown) displaying operators other than the user interface 50. By using the screen switching operator 57, it is possible to select more operators.

[0065] The user interface 50 also includes a pointer P for operating / selecting each of the above-mentioned controls. The pointer P is displayed on the display unit 210 based on the line-of-sight information acquired by the acquisition unit 201. Specifically, the position of the point of gaze on the screen of the display unit 210 is obtained from the line-of-sight information, and the pointer P is displayed at that position.

[0066] Fig. 6 is an enlarged view of the pointer P shown in Fig. 5. As shown in Fig. 6, the pointer P has a reference position P1 and a circular area P2 of a predetermined size that is approximately centered on the reference position P1. An index Q is also displayed near the circular area P2 of the pointer P. The index Q indicates the beat and bar number of the music piece being played (the sound source being played) in the format "000.00 (beat.bar number)".

[0067] Furthermore, the display control unit 203 can enlarge and display a predetermined range including the pointer P of the user interface 50. This allows the operation / selection of each operator to be performed more reliably. Furthermore, since an indicator Q is displayed near the pointer P, even when the display is enlarged, the user can operate each operator while always checking the beat and bar number of the music being played.

[0068] Returning to FIG. 3 , the determination unit 204 determines the selection of an operator that overlaps with at least a part of the pointer P from among the plurality of operators. This allows the selection of an operator to be determined solely through gaze input using the eye tracking device 21. The determination unit 204 may also be configured to determine the selection of an operator that overlaps with the reference position P1 of the pointer P. This reduces the risk of erroneous operation (erroneous selection) of an operator when operators are arranged adjacent to each other. The determination unit 204 may also be configured to determine the selection of an operator that has been overlapping with at least a part of the pointer P for a predetermined time (e.g., two seconds). This allows the selection of an operator to be determined more reliably. This selection method allows the selection of an operator to be determined solely through gaze information, and is therefore useful for users who have difficulty performing actions other than moving their gaze.

[0069] Furthermore, the determination unit 204 may limit the operators to be selected in response to switching of the sound source, depending on the fader operators (crossfader operator 53 and fader pattern operator 54). That is, the determination unit 204 may determine the selection of an operator included in the first operator group when the volume of the first sound source is larger than the volume of the second sound source, and may determine the selection of an operator included in the second operator group when the volume of the second sound source is larger than the volume of the first sound source. This makes it possible to prevent erroneous operation of an operator for the first sound source (first operator group 51) while the second sound source is being played back.

[0070] Furthermore, the determination unit 204 may determine the selection of an operator in accordance with an input operation by the user using the input device 22, which is received by the receiving unit 202. That is, the user can select an operator by gaze input using the eye tracking device 21, and then determine the selection of the operator using the input device 22 (a manual switch, a foot switch, or the like). This allows the user to determine the selection of an operator more quickly than when determining the selection of an operator that has been overlapping with at least a part of the pointer P for a predetermined period of time. This selection method is useful for users who can operate the input device 22 with parts of their body, such as their hands or feet, in addition to moving their gaze.

[0071] The storage unit 211 stores information relating to each control command of the musical sound generating device 3 (DAW) associated with each operator.

[0072] The communication unit 212 transmits to the musical sound generating device 3 information (control command information, MIDI event information) conforming to the MIDI standard corresponding to the operator selected by the determination unit 204 .

[0073] Next, we will explain the functional configuration of the musical sound generating device 3. As shown in Fig. 3, the musical sound generating device 3 has an audio input unit 301, a sequencer 302, a recorder 303, a sound source 304, a mixer 305, a sound system 306, and a communication unit 307.

[0074] The audio input unit 301 receives input of an analog audio signal converted into a digital form, or inputs a digital audio signal itself.

[0075] The sequencer 302 records the MIDI event information received by the communication unit 307. This MIDI event information is supplied to the sound source 304, which will be described later.

[0076] The sound source 304 is a software sound source. The sound source 304 generates an audio signal in response to input MIDI event information. The audio signal output from the sound source 304 is input to the recorder 303 and the mixer 305.

[0077] The recorder 303 records the audio signal input by the audio input unit 301 and the audio signal output from the sound source 304. The audio signal recorded in the recorder 303 is output to the mixer 305 in response to a playback instruction.

[0078] The mixer 305 supplies the audio signals output from the sound source 304 and the recorder 303 to the sound system 306 .

[0079] (Example of operation) Next, a description will be given of an example of the operation of the musical sound generating system 1. Fig. 4 is a flowchart showing an example of the operation of the musical sound generating system 1.

[0080] First, the acquisition unit 201 of the control unit 200 in the controller 2 acquires the user's gaze information output from the eye tracking device 21 (step S1). The gaze information may be acquired at any timing, for example, constantly in real time.

[0081] Next, the display control section 203 of the control section 200 displays a plurality of operators associated with the control commands of the musical sound generating device 3 (DAW) on the display section 210 (step S2).

[0082] Furthermore, the display control unit 203 of the control unit 200 displays a pointer for operating the operator on the display unit 210 based on the line-of-sight information acquired by the acquisition unit 201 (step S3).

[0083] Next, the determination unit 204 of the control unit 200 determines the selection of an operator that overlaps at least a part of the pointer from among the multiple operators (step S4).

[0084] Next, the communication unit 307 transmits information conforming to the MIDI standard corresponding to the operator selected by the determination unit 204 to the musical sound generating device 3 (step S5).

[0085] Then, the musical sound generating device 3 performs various processes (execution of control commands, recording of MIDI events, etc.) based on the information conforming to the MIDI standard received from the controller 2 (step S6).

[0086] As described above, according to this embodiment, the musical sound generating system 1 comprises an acquisition unit 201 that acquires the user's gaze information, a display control unit 203 that displays on the display unit 210 a plurality of operators associated with the control commands of the musical sound generating device, and displays on the display unit 210 a pointer P for operating the operators based on the gaze information acquired by the acquisition unit 201, a determination unit 204 that determines which of the plurality of operators overlaps with at least a part of the pointer P, and a communication unit 212 that transmits information conforming to the MIDI standard corresponding to the operator selected by the determination unit 204 to the musical sound generating device 3, and the plurality of operators include at least one fader operator (crossfader operator 53 and / or fader pattern operator 54) that switches the sound source being played from a first sound source to a second sound source, so that a user with limited physical movement can easily control the musical sound generating device 3.

[0087] Any part or all of the functional units described in this specification may be realized by a program. The program mentioned in this specification may be distributed by being non-temporarily recorded on a computer-readable recording medium, or may be distributed via a communication line (including wireless communication) such as the Internet, or may be distributed in a state where it is installed on any terminal.

[0088] Based on the above description, a person skilled in the art may be able to conceive additional effects and various modifications of the present invention, but the aspects of the present invention are not limited to the individual embodiments described above. Various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and spirit of the present invention, which is derived from the content defined in the claims and their equivalents.

[0089] For example, what is described herein as a single device (or component, the same applies hereinafter) (including what is depicted as a single device in the drawings) may be realized by multiple devices. Conversely, what is described herein as multiple devices (including what is depicted as multiple devices in the drawings) may be realized by a single device. Alternatively, some or all of the means and functions included in a certain device (e.g., a server) may be included in another device (e.g., a user terminal).

[0090] Furthermore, not all of the features described in this specification are essential requirements. In particular, features described in this specification but not included in the claims can be considered optional additional features.

[0091] It should be noted that the applicant is merely aware of the inventions disclosed in the documents listed in the "Prior Art Documents" section of this specification, and the present invention does not necessarily aim to solve the problems of the disclosed inventions. The problem that the present invention aims to solve should be determined by taking into consideration the entire specification. For example, if this specification states that a specific configuration achieves a certain effect, it can also be said that the present invention solves a problem that is the reverse of that effect. However, it is not necessarily intended that such a specific configuration be an essential requirement. [Explanation of symbols]

[0092] 1. Musical tone generation system 2 Controller 21 Eye tracking device 22 Input Devices 201 Acquisition Department 202 Reception 203 Display control unit 204 Decision Section 210 Display section 211 Storage section 212 Communications Department 3. Musical tone generator

Claims

1. A controller for controlling a musical sound generating device, an acquisition unit that acquires user's line of sight information; a display control unit that displays a plurality of operators associated with the control commands of the musical sound generating device on a display unit, and displays a pointer on the display unit for selecting the operators based on the line-of-sight information acquired by the acquisition unit; a selection unit that selects an operator that overlaps at least a part of the pointer from among the plurality of operators; a communication unit that transmits information conforming to the MIDI standard corresponding to the operator selected by the determination unit to the musical sound generating device, The controller, wherein the plurality of operators include at least one fader operator for switching the sound source being reproduced from a first sound source to a second sound source.

2. 2. The controller of claim 1, wherein the at least one fader operator includes a crossfader operator including a shaft and a knob, and the crossfader operator continuously changes the volume of each of the first sound source and the second sound source depending on the position of the knob on the shaft.

3. 2. The controller according to claim 1, wherein the at least one fader operator includes a fader pattern operator for continuously changing the volume of each of the first sound source and the second sound source according to a pre-stored pattern.

4. The controller according to claim 1 , wherein the plurality of operators include a first group of operators for operating the first sound source and a second group of operators for operating the second sound source.

5. 5. The controller according to claim 4, wherein the display control unit highlights the first operator group when a volume of the first sound source is greater than a volume of the second sound source, and highlights the second operator group when a volume of the second sound source is greater than a volume of the first sound source.

6. 5. The controller according to claim 4, wherein the determination unit determines the selection of an operator included in the first operator group when a volume of the first sound source is greater than a volume of the second sound source, and determines the selection of an operator included in the second operator group when a volume of the second sound source is greater than a volume of the first sound source.

7. the first operator group includes a sound source selection operator for selecting an odd-numbered sound source from among a plurality of sound sources to be played back consecutively; 5. The controller according to claim 4, wherein the first operator group includes a sound source selection operator for selecting an even-numbered sound source from among a plurality of sound sources to be played back consecutively.

8. the first operator group is displayed on the left side of the display unit, The controller according to claim 7 , wherein the second group of operators is displayed on the right side of the display unit.

9. The controller of claim 8, wherein the plurality of operators include a plurality of clip selection operators for selecting a video clip to display during playback of an audio source, and the plurality of clip selection operators are displayed between the first group of operators and the second group of operators.

10. The controller according to claim 4 , wherein the first operator group and the second operator group include an effect operator for applying an effect to a sound source being reproduced.

11. The controller according to claim 4 , wherein the first group of operators and the second group of operators include an XY pad capable of simultaneously changing two types of parameters.

12. The controller according to claim 1 , wherein the plurality of operators include a screen switching operator for switching to a screen on which another operator is displayed.

13. a receiving unit that receives an input operation by a user using an input device; The controller according to claim 1 , wherein the determination unit determines the selection of the operator in accordance with an input operation received by the reception unit.

14. the input operation is pressing a switch with a part of the user's body; The controller according to claim 13 , wherein the determination unit determines the selection of the operator based on the user's line of sight information and without any user operation other than pressing the switch.

15. The controller according to claim 1 , wherein the determination unit determines the selection of an operator that has been overlapped with at least a part of the pointer for a predetermined period of time.

16. The controller according to claim 1 , wherein the determination unit determines the selection of an operator that overlaps with the reference position of the pointer.

17. The controller of claim 16 , wherein the reference position is approximately the center of the pointer.

18. The controller according to claim 1 , wherein the display control unit displays beats and bar numbers of the music being played near the pointer.

19. The controller according to claim 1 , wherein the determination unit determines the selection of the operator without any user operation other than the user's line of sight information.

20. A musical sound generating system comprising a musical sound generating device and a controller for controlling the device, an acquisition unit that acquires user's line of sight information; a display control unit that displays a plurality of operators associated with the control commands of the musical sound generating device on a display unit, and displays a pointer on the display unit for selecting the operators based on the line-of-sight information acquired by the acquisition unit; a selection unit that selects an operator that overlaps at least a part of the pointer from among the plurality of operators; a communication unit that transmits information conforming to the MIDI standard corresponding to the operator selected by the determination unit to the musical sound generating device, The musical sound generating system, wherein the plurality of operators include at least one fader operator for switching the sound source being reproduced from a first sound source to a second sound source.

21. 1. A method for controlling a musical tone generating device, comprising: acquiring user's gaze information; displaying a plurality of operators associated with control commands of the musical sound generating device on a display unit, and displaying a pointer on the display unit for selecting the operators based on the line-of-sight information; determining a selection of a control from the plurality of controls that overlaps at least a portion of the pointer; transmitting information conforming to the MIDI standard corresponding to the determined operator to the musical sound generating device; The method, wherein the plurality of controls includes at least one fader control for switching a currently playing sound source from a first sound source to a second sound source.

22. A program for causing a computer to execute a method for controlling a musical sound generating device, the method comprising: a step of displaying a plurality of operators associated with control commands of the musical sound generating device on a display unit, and displaying a pointer on the display unit for selecting the operators based on user line-of-sight information; determining a selection of a control from the plurality of controls that overlaps at least a portion of the pointer; transmitting information conforming to the MIDI standard corresponding to the determined operator to the musical sound generating device; The program, wherein the plurality of operators include at least one fader operator for switching a sound source being reproduced from a first sound source to a second sound source.

23. A computer-readable storage medium storing the program according to claim 22.

Images