Music composition and production controller

Abstract

A music input system comprising a gesture sensing device and a controller for generating a gesture interface zone 22, for example for a touchscreen or other gesture sensing device, such as an eye / head

Description

The present invention relates to controls for the creating and playing music on electronic devices, such as, for example, an input device for controlling music production software. The applicant’s earlier patent application, published as WO 2019 / 234424 A1, addresses certain issues surrounding the accessibility of music production software. That disclosure proposes the use of a bespoke input device comprising a joystick and buttons to allow users to play music using the device as well as control certain music production functions of associated software running on a connected computer. The joystick arrangement provides an intuitive and precise tactile input device that can be used by novices and experts, including users with physical impairments, to great effect. The input device can be used to replace other conventional instruments or input devices, such as keyboards, when teaching music, producing or performing music. However, the need for bespoke physical input devices provides a barrier to entry for some users and can mean that large number of devices are needed, for example in classroom settings, if it is desired to teach a number of students concurrently. Furthermore, the global pandemic altered attitudes associated with teaching music in person with many students choosing to learn to play musical instruments or take music production courses online. Separately from tools for teaching / learning music, there are also video games, e.g. so-called rhythm-action games, in which a user is challenged with matching sequences of inputs to music. However, conventional games of this type are aimed at timing of non-musical inputs, i.e. gaming inputs of a different kinds, to fit the rhythm or tempo of music that is played in the background. As such, rhythmaction games do not typically provide creative tools for the player to express themselves for the purpose of composing music. In other music-themed video games, e.g. which are aimed at recreating a music performance, the user requires bespoke hardware or reuses conventional video game controllers to provide music inputs. However, again, these controllers do not allow high levels of musicality / creativity and instead provide a simplified user device to capture the timing of a player’s control input, e.g. to trigger playback of a pre-recorded note / chord with the challenge of matching an instrumental track of an existing song. The present invention has arisen from ongoing development of the original concepts disclosed in WO 2019 / 234424 A1 in order to provide an input device and associated systems that can mitigate one or more of the above-discussed problems. It may be considered an aim of the invention to provide an input device or system for music creation that is accessible / intuitive whilst offering a high level of musicality. Statements of Invention According to a first aspect of the invention, there is provided a music input system comprising a gesture sensing device and a controller comprising machine readable instructions for generating a gesture interface zone to receive user inputs for sensing by the gesture sensing device, the controller processing sensed user gestures relative to the interface zone to generate an output signal, wherein gesture interface zone has a plurality of regions recognised by the controller and the controller assigns different music notes or chords to user gestures within the different regions of the gesture interface zone and wherein the controller identifies one or more movement parameter for the sensed user gesture, the controller appending a characteristic to the assigned music note or chord based on said movement parameter, the controller generating the output signal comprising data identifying the assigned music note or chord and the appended characteristic. The terms ‘notes’ and ‘chords’ as used herein may cover individually played notes / chords, sequences of notes / chords and / or the stream of data or signal associated therewith. The music input system may be an electronic musical instrument or controller. The music input system may comprise software / firmware for capturing notes / chords played on the electronic musical instrument by the user. According to a second aspect of the invention there is provided a data carrier comprising machine readable instructions for operation of one or more processor of a gesture sensing device to: define a gesture interface zone for receiving user inputs; identify a plurality of different regions of the gesture interface zone and assign different music notes or chords to the different regions of the gesture interface zone; monitor one or more movement parameter for user gestures sensed within the gesture interface zone by the gesture sensing device; determine a characteristic for the assigned music note or chord based on said monitored movement parameter; and generate an output signal comprising data identifying the assigned music note or chord and appended playback feature. The characteristic may determine one or more quality of the assigned music note or chord, e.g. how the note / chord sounds. The characteristic may comprise a playback characteristic or feature of the note / chord. A plurality of characteristics may be assigned, e.g. to an individual note / chord or to each note / chord. The processing to generate the output signal is preferably performed on the device such that the output signal can be output to a music playback or production device / system. The output signal may be generated according to a predetermined signal format or content, e.g. as is recognisable to other music production or playback devices. The output signal may comprise a Musical Instrument Digital Interface (MIDI) signal or a development / equivalent of the MIDI format. The gesture sensing device may comprise tactile interface, such as a touchscreen device. Additionally or alternatively, the gesture sensing device may comprise a vision, imaging or camera device or non-contact gesture sensing device. An eyetracking, face-tracking, hand-tracking or body-tracking sensor system could be used. The interface zone may or may not be two-dimensional. The regions may be areas within the zone. The regions may be adjacent or adjoining. The controller may determine the location of initiation of a user gesture within the interface zone. The controller may determine the location of user contact on the touchscreen. The user contact with the touchscreen may be determined according to a contact area, e.g. with a digit of a user. A contact point or spot may be determined for the contact area, e.g. a determined contact point within the contact area. The central point may be used to determine where contact with the touchscreen is deemed to have occurred by the controller / processor. The different regions of the interface zone and / or interface zone itself may be defined according to a cartesian coordinate system. The plurality of regions may be predetermined (e.g. relative to the interface zone) or may be determined reactively, e.g. in response to a user gesture. The interface zone and / or plurality of regions may be generated / defined according to a location of user contact (e.g. with a touchscreen) or another initiating gesture. The user contact or initiating gesture may be used to define a centre or central region of the interface zone. The plurality of regions of the interface zone may be discreet and / or adjoining. The plurality of regions may be contained within a periphery of the interface zone. The interface zone and / or plurality of regions may be displayed to the user via a display. The different regions may be defined in the memory of the device. The touchscreen may have a continuous surface with the different regions thereof defined in the memory of the device and displayed on the touchscreen. At least four, six or eight regions of the touchscreen may be defined, each being assigned to a different musical note / chord. The regions may represent the notes of an octave. The movement parameter may comprise one or more of a direction, speed, acceleration, duration or distance of a gesture, e.g. by movement of the gesture through / within one or more zone. The movement parameter may be determined during contact with the touchscreen. One or more contact points may be input by the user or processed by the controller concurrently. A corresponding plurality of notes / chords may be processed by the controller concurrently. Initiation of the gesture / contact within the interface zone, e.g. within one of the regions, may cause a note on event. Cessation of the gesture / contact, or movement of the gesture beyond the interface zone or region, may cause a note off event. Note duration may be determined as the elapsed time or distance between note on and note off events. The controller may determine a velocity value for the assigned music note based upon the speed or acceleration of movement within the interface zone or a region thereof, e.g. the speed / acceleration of the point of contact with the screen or the speed / acceleration of a gesture mapped to be within a region of the interface zone. The controller may determine a direction of movement of a gesture within the interface zone or a region thereof. The controller may determine a change, and / or rate of change, in direction of movement of a gesture within the interface zone or a region thereof. The controller may process the direction, change in direction or rate of change in direction to determine one r more quality or attribute of a note / chord being played. The movement parameter and / or user contact may be determined for a single gesture (e.g. a continuous gesture) and / or a single instance of contact by a user, e.g. a single continuous instance of user contact. In the manner described herein, the location and / or movement of a gesture through an interface zone may be used to create music notes / chords or sequences with a variety of different qualities. This tactile or gesture-based approach means it is simple and intuitive for users to engage with the music creation process, whilst offering significant depth of expression for more experienced users. Music notes / chords to be played may be communicated to the user as directional gesture inputs, e.g. symbols in the form of arrows. The angle / orientation of the arrow may denote / determine the note selection. Additionally or alternatively, a colour of the arrow / symbol may denote / determine the note selection. The length or thickness of the arrow may determine one or more movement parameter to be applied as a gesture by the user. The arrows may additionally / alternatively be colour coded with different colours either applying to different notes or different playback features for the notes. The gesture inputs may be communicated to the user on screen, on another visual / digital display or as printed / paper material. Thus, as well as providing a control scheme for playing / recording music, the invention may provide a novel method of instructing / teaching users to use an electronic input device. Unlike conventional instruments, which require the user to be able to read musical notation, the invention allows sequences of notes to be communicated as simple directional gesture inputs to the user, e.g. on screen or in printed format. Furthermore, unlike some simplified video game input devices, the invention allows pitch and / or musical characteristic in put at the time of note selection using a common interface. The user input gesture may convey starting and ending times, pitch (e.g. in addition to note selection) and loudness for a note / chord. The controller may monitor the user input gesture to derive values for any / all of those parameters. An instrument selection may also accompany the note / chord information. One or more further characteristic (e.g. a playback feature or quality) for the assigned note / chord may be determined according to a further aspect of the user input gesture, e.g. a contact parameter, movement parameter or other transient change in the gesture direction, contact area, etc. The further feature / quality of the note may be determined or applied by the controller. For example, the pressure with which a user contacts the touchscreen, e.g. determined according to contact pressure or contact area, or the direction of movement through the region of the interface zone may be used to determine a different quality of the note that would affect its playback. In this manner additional qualities of the note, such as aftertouch, note attenuation or pitch bend could be accommodated all within the gesture-based system. One or more characteristic may be applied at different times for the duration of a note / chord being played. One or more characteristic may or may not be applied during an initial portion of a note / chord being played. One or more characteristic may or may not be applied during a main or middle portion of a note / chord being played. One or more characteristic may be applied during a final portion of a note / chord being played. The initial or final portion may represent an initial / final portion of a user gesture, e.g. a fifth, quarter or third of the note / chord or gesture. The middle portion may represent a similar or different fraction / portion of the note / chord or gesture, such as larger portion, e.g. a middle third, half or two-thirds. A continuous control (CC) value or CC data may accompany the note / chord information, e.g. to apply the characteristic. The CC data may be applied by the controller. The CC data may be derived by the controller from the user gesture input or another user input. Different CC values may be applied to different portions of the user gesture input. The gesture may be bi-directional, e.g. radially outward or inward relative to a centre of the interface zone. Notes / chords can be initiated / played by gestures in either direction. According to a third aspect, there is provided a computational device comprising the data carrier according to the second aspect. According to a further aspect of the invention, there is provided a method of playing music using the music input device according to the first aspect, the data carrier of the second aspect or the computational device of the third aspect. Any essential or optional feature described in relation to one aspect of the invention may be applied to any further aspect. Detailed Description Working embodiments of the invention are described in further detail below by way of example only with reference to the accompanying drawings, of which: Fig. 1 shows an example user interface zone for a user input device; Fig. 2 shows an example user input device with an interface zone on screen; Fig. 3 shows an example graphical user interface for a user input device; Fig. 4 shows an example flow of steps according to one example of operation of a user input device; Fig. 5 shows an example gesture input within the user interface zone; Fig. 6 shows an example of ancillary user interface controls accompanying the user interface zone; Fig. 7 shows an example tuning arrangement for the user interface zone; Fig. 8 shows an example of further information that may accompany a user gesture input; Fig. 9 shows an example of an adaptive user interface zone; Fig. 10 shows an example of a further user interface having a plurality of zones; Fig. 11 shows an example display for monitoring head or eye tracking within the interface zone; Fig. 12 shows an example flow chart of processing steps for head / eye tracking implementation of user gesture control; Fig. 13 shows a schematic of eye tracking implementation; Fig. 14 shows a schematic of an example virtual reality or augmented reality implementation of the user interface zone; Fig. 15 shows a three-dimensional cartesian coordinate system for implementation of one or more further degree of freedom for user gesture input. Turning firstly to Figs. 1 and 2, there is shown an example implementation of a user interface within the context of a touchscreen device 10. Whilst the user tactile interface and display screen are integrated in this example, they could be separate, e.g. in the form of a ‘slate’ or other touch-sensitive surface which is separate from the display screen. The device 10 has a touch-sensitive screen 12 of conventional type (e.g. a capacitive or resistive touchscreen) on which a user display is provided so as to define a visual indicator of a user interface zone 14. The user interface zone is generated and monitored by a device controller, which typically takes the form of one or more computer processors operating under machine readable instructions in accordance with embodiments of the invention. The interface zone 14 is rectangular in this example but could alternatively be circular or polygonal if desired. The interface zone 14 defines an area in which user inputs in the form of gestures will be monitored by the device controller and processed in order to generate output signals in the form of notes / chords as will be described below. The device 10 can thus be used as a music input device or instrument for playing / recording music in the form of a sequence / composition of notes, such as a riff, track, score or the like. A plurality of radial directions 15a-h are shown in Figure 1. These different directions are equally angularly spaced, e.g. about a centre of the interface zone 14. The directions 15a-h denote a location, direction or angular orientation associated with a musical note / chord. There are eight directions, each assigned to a different note of an octave. The user interface zone 14 is divided into regions or areas 14a-14h as shown in Figs. 1 and 2. The regions 14a-h are assigned to each direction 15a-h. That is to say each region 14a-h has the associated direction 15a-h at its centre. The regions assist in processing of user inputs to select notes and characteristics of notes. The regions 14a-h are adjoining and are arranged at different angular orientations. The regions 14a-h extend from a common central point radially outwardly. The regions 14a-h widen with distance form the centre point and may be considered to be akin to segments of the user interface zone 14. Each region 14a-h is assigned a music note of chord by the controller (i.e. according to its direction 15a-h) such that a gesture occurring within the region identified by the controller will cause the controller to output a music note / chord according to the region in which the gesture occurred, e.g. in which the gesture starts, ends or passes through. The examples described herein make use of the MIDI format and each of regions 14a-h has a MIDI note (and a CC value) associated with it. At the centre of the user interface zone and / or the regions 14a-14h, there is a dead zone 16, i.e. a region / area that is not assigned to a music note / chord and will not therefore trigger processing of a note / chord output signal by the controller. Additionally or alternatively, the central / dead zone 16 may be used to access further functions, e.g. a menu or selection interface for altering a type or style of note, chord or sequence. When the software application for playing music is running, the controller generates the user interface zone according to a cartesian coordinate system. Thus the x,y coordinates associated with the dead zone 16 and each of the regions 14a-h can be established within the user interface zone. The contact between a user’s digit and the interface zone can thus registered with respect to the cartesian coordinate system to determine where it lies with respect to the interface zone. As shown in Fig. 2, the contact may be identified as an area of contact 18. For greater accuracy, the area of contact 18 may be processed to establish a point of contact 20, which may be used by the controller for the purpose of determining an output musical note / chord signal. The location, and velocity (e.g. direction and speed) of the point of contact 20 can be processed as described herein. In Fig. 2 the different regions of the user interface are represented as shaded areas with differing contrast. However they may be presented to the user in a variety of ways, e.g. using visual cues or indica to denote different notes / regions. Turning to Fig. 3, there is shown an example of an actual user interface 22. In the user interface 22, there are a plurality of arrows or chevrons 24a-h displayed in the interface zone in order to represent the directions 15a-h and / or regions 14a-h. The arrows 24a-h are radially spaced from the central dead zone. In this example, the arrows are colour coded. The arrows 24a-h provide indica for a user of a direction to be played using the interface. Thus the arrows 24a-h can be used to represent musical note to be played by a user in a music notation system. In this way, a user does not need to be able to read musical notation or sheet music in order to play a specific song. Instead the order and timing of notes to be played can be communicated as a sequence of arrows, which a user can follow. The sequence of arrows could for example be displayed on screen, e.g. with each arrow being displayed successively at the time it is to be played by the user. Alternatively multiple arrows could be displayed in order for the user to follow, or else in an animated whereby arrows move along the screen to an indicated corresponding to the time each note is to be played. In other examples, arrows could be provided in a printed format. The applicant has coined the term ‘Arrow Notes’ for this simple system by which notes can be communicated to a user to play. A central icon 25 is displayed to represent the centre of the interface zone, i.e. aligned with the dead zone 16. Where the placement of the central zone is defined when a user touches / interacts with the interface, the note on signal may be received / determined after a movement of a predefined (or user changeable) number of pixels, degrees or other threshold value to denote movement by the user. Within this dead zone, all velocity and note information may be calculated by the controller before initiating playing of the note, i.e. before sending the note on or control data message for note capture / playback. Accompanying the regions of the interface zone (i.e. the arrows), there are a plurality of buttons 26 provided as part of the user interface, e.g. on screen. The buttons 26a and 26b allow an increase or decrease in octave, i.e. allowing a user to easily move up or down an octave in a button press. Other buttons 26c-f allow effects or other characteristics to be applied to notes as they are played. These could be default controls / effects or else could be assigned by a user. Tap Player The Tap Player mode provides a particularly simple way of interacting with the interface. When a user taps inside the user interface zone, the direction and velocity of the tap are calculated based on the coordinates relative to the centre of the interface zone. The device controller sets the interface zone as described above according to the available size of the screen, or relevant portion thereof - such as a window. The controller then establishes the cartesian coordinate system for the available area as the interface zone and a central point of the interface zone. When a user taps the screen, an algorithm determines one of the eight radial directions from the central zone corresponding to the arrow note, and the velocity of the note is adjusted based on the distance of the tapping coordinate from the centre of the control. Specifically, the algorithm outputs a direction that aligns with the arrow 24, providing a visual and intuitive representation of the chosen musical element. The velocity of the note is dynamically adjusted, with higher velocities associated with tapping coordinates further away from the centre of the control. The user may tap the screen quickly or maintain contact for longer. A note on signal may be identified upon first contact with the screen within one of the regions 14a-h. A note off signal may be generated when the user removes their digit from the screen. Thus note duration can be dynamically adjusted by the user. This design enhances user expressiveness and allows for a nuanced musical output. In addition to the tap player's core functionality, the various controls are integrated to enrich the musical experience. These controls include sustain, chords, chords extension, chord inversion, and octave adjustments, which are easily accessible through the buttons 26a-f displayed under the tap player interface, allowing users to explore and customize their musical expressions seamlessly. The tap player, with its intuitive design and inclusive features, caters to users with different levels of musical proficiency and physical abilities. The combination of the ‘Arrow Note’ control and additional functionalities enhances the versatility and accessibility of the system. Swipe Player The Swipe Player mode allows a distinctive and expressive control style within the input device. This control mode utilizes the Arrow Note interface zone as a touchprocessing area, detecting directional swipe gestures made inside it and outputting note / chord signals accordingly. To engage the Swipe Player, users perform swipe gestures within the Arrow Note interface zone. The algorithm analyses the direction of the swipe gesture (i.e. the moving contact point between the user’s digit and the screen) in real-time and triggers the note when a certain distance is travelled from the start position of the swipe. The velocity of the note is dynamically calculated based on the speed and direction of the swipe gesture, allowing users to achieve nuanced musical expressions. Importantly, the size of the swipe control adjusts based on the physical display size of the device, ensuring adaptability to various screen dimensions. This adjustment is seamlessly integrated into the algorithm, enhancing the overall user experience across different devices. The process followed by the controller is shown in Fig. 4. Upon initiation of the software application, the controller establishes the interface zone 14 and the associated cartesian coordinate system as described above, including the dead zone 16 and the regions14a-h. When a user touches the interface, the controller determines the location of the contact point at the initiation of the touch. If the contact point is within a region 14a-h the controller determines a note on event and follows the processing steps shown in Fig. 4. If the contact point is initially outside of the regions 14a-h, the controller monitors the swipe gesture for the point at which it enters one of the regions 14a-h, at which point it determines a note on event. Turning to Fig. 5, the is shown an example swipe gesture. In this example, the initial contact point of the swipe gesture 28 is within the dead zone 16 and therefore does not trigger a note on event. As the point of contact travels through the dead zone 16, the controller can commence processing the movement of the contact point. When the contact point crosses the interface between the dead zone 16 and a region 14a-h, the controller initiates a note on signal. The controller selects the note assigned to direction 15h and one or more attributes of the note. The timbre or instrument may be previously selected. The velocity is determined according to the velocity of the swipe (or the speed of the swipe in the radial direction assigned to that note, which may be a component of the actual direction of the swipe direction). When the user removes their digit from the screen or exits the selected region, the controller determines a note off event. At that point the entire data for that note / chord can be logged and saved as a single note / chord or a part of a sequence. I the example of Fig. 5, a second part of the swipe gesture is a return gesture towards the centre, i.e. radially inwardly. As that gesture enters the region 14g in direction 15g, the note assigned to that direction / region is selected. On return to the centre, a note lock event is triggered, e.g. as shown by line or threshold 30. It can be understood that the gestures, radially outwardly or inwardly convert to inputs for playing notes / chords. Fig. 6 shows how the on screen buttons can be used to alter the note being played. Some buttons may be activated before a note is played or during the swipe gesture, i.e. during the note event, depending on their function. They can change the type and amount of playable notes as well as adding characteristics of qualities to the notes such as sustain, timbral changes, filters, equalisation and / or saturation. Fig. 7 shows an example note assignment or tuning scheme for the interface. Fig. 8 shows how a gesture movement can be converted to MIDI or other equivalent or similar music data formats. As well as the aforementioned, note on, note off and velocity data being assigned to the note (i.e. to determine note duration and velocity), further information is conveyed by the gesture itself or the selection of buttons to alter the note. For example, the gesture can change direction during traversal through a region of the interface zone, i.e. rather than following a straight line radially, which can be used to affect a quality of the note. The user can remove their digit, hold it still in a region, or exit the selected region, which each may have an impact of the information associated with the note. In one example, the tap gesture and swipe gesture could be combined. For example an initial tap may be akin to an impulse mimicking for example a piano key or pluck of a guitar string, followed by a swipe gesture to affect the ongoing note. The pressure of the initial tap could be monitored as well as, or in addition to, the velocity of the swipe gesture. A distinction could be made for example between an initial tap in a region of the interface zone and a swiping gesture to enter a region. A swiping gesture may be more akin to a woodwind, brass, string or synthesised instrument or the like, for which there may be a reduced initial impact. Turning to Fig. 8, it can be seen that the note / chord could have attributes for its initiation (e.g. an initial portion / third), attributes assigned to it cessation (e.g. a latter portion / third) and / or attributes applied to the whole swipe gesture. A curve or profile for the note may be selected to affect the quality of the note, e.g. as a whole. A velocity profile or other attribute could be selected to match a desired curve / profile. Therefore, as well as note data the user inputs can be processed to generate control data which can be formatted in multiple ways dependent on user selections. With reference to Fig. 8, such a control scheme may comprise: Expression -1 MIDI CC across all directions Control -1 MIDI CC per each direction Expressive Control - both of the above settings at once FX - 4 MIDI CC values that overlap to create nested XY pads Bottom - adds a CC 0-127 on the bottom 3rd of the expression setting Top - adds a CC 0-127 on the top 3rd of the expression setting This is just one example and there are multiple implementations of MIDI CC and note information being sent by a single gesture that could be developed based on the principles described herein whilst the available control options are explored and developed. Although the embodiments herein are described in relation to MIDI data, it will be appreciated that alternative data formats could be used that are developments of MIDI, alternatives to MIDI or equivalent to MIDI without departing form the scope of the invention defined by the claims. In some examples, the output signal form the user device could be in an interim or bespoke data format comprising all the digital information needed for recreation of musical notes, chords or scores, etc. but which can later be converted by external computational means into a more relevant or widely-used data format. The Swipe Player provides a tactile and immersive musical interaction, offering users a control style that closely mirrors the experience of a variety of different musical instruments. This expressive input method, combined with the dynamic analysis of swipe gestures, contributes to the versatility and authenticity of the system to allow it to be used in a manner unlike previous electronic input devices. It caters to users who appreciate a more interactive and physically engaging approach to music creation, fostering a deeper connection between the user and the creative process. It can also offer deeper levels of user interaction or musicality, for example in a rhythm or music-based game. In Fig. 9 it is shown how the interface can be reactive i.e. being established in response to a user’s first contact with the screen. To automatically set up the interface zone with its centre at the initial point of contact. In this manner the user may not need to accurately position their digit on a pre-defined interface zone but instead the note can be played entirely based on the swipe gesture itself, e.g. from anywhere in an existing interface. In Fig. 10, there are shown a pair of interface zones so that a user can play notes concurrently or in quick succession, e.g. with different digits or hands. The two or more interfaces could be assigned to different octaves or instruments for example. Additionally or alternatively, one interface zone could be used for note selection and velocity, with the other interface being used to add a further quality to the note, e.g. akin to aftertouch or similar. The controller can thus process the two interfaces separately or collectively as necessary according to the user’s preferences. Further Implementations Figs. 11 to 14 show how the same principles and processing defined above for a touch screen can be used to implement other control schemes. In Figs. 11 and 12, the implementation scheme is based on head tracking and movement, i.e. using a camera as an input, rather than relying on a swipe and / or tap gesture. In Figs 11a-c it can be seen that the same interface area can be established and the movement of a user’s head can be mapped to the directions 15a-h or zones 14a-h described above. The controller can define all the note data described above in the same way as for a swipe but based on the orientation and movement of the user’s head. Thus to play a note, the user moves their head in the direction of one of the notes / chords at a speed indicative of the velocity of the note. A different gesture, in this example, as smile, can be used to pull up the options menu of Fig. 11c before playing a note or sequence of notes. The selected attributes will then be applied when the user returns to the interface zone display to play the notes / chords. The steps taken by the controller to process head movement, e.g. tilt and roll, are shown to confirm how the head gestures can be converted to map onto the interface zone and then output note data accordingly. It is also noteworthy that different gestures, e.g. a blink or eyes closed / open, can be used to add extra information as shown in Fig. 16. For example, a user closing their eyes during playing a note may add an effect that has been predefined or selected by a user. Alternatively, a user closing their eyes could be used to end the duration of a note. For example a note duration may be applied only when eyes are open, e.g. to mimic the Tap Player or functions of the Swipe Player described above. In Fig. 13 there is shown that eye-tracking could be used instead of head tracking to mimic the swipe gestures or tap gestures described above. Thus eye tracking could be used, without the need to process the whole head movement if desired any may allow for a more responsive or accessible control scheme for certain users. It will be appreciated that eye tracking camera applications, e.g. on front facing cameras of smart phones, tablets or PCs, have been implemented for a number of years and can usefully be applied in real time for the current application to playing / composing music. In Fig. 14, there is shown a potential virtual reality implementation of the technology. The user interface may be displayed in a virtual reality or augmented reality context to the user. Conventional VR systems allow either hand tracking or tracking of controllers held by a user, which can then be mapped to a user interface zone in the same way as the swiping gestures described above. In this manner a mid-air hand gesture and / or mid-air haptics could be used to control the playing of notes / chords and / or provide haptic feedback to a user. Additionally or alternatively, the head tracking (e.g. using accelerometers / gyroscope technology in the headset) or eye tracking (e.g. using onboard imaging systems) can be used to implement those modes of operation in place of hand-based gestures if desired. The VR / AR implementation can allow a potentially more seamless gesture-based instrument that feels intuitive to play and allow a different feeling for the user than conventional instruments that require interaction with a physical piece of hardware to capture the user inputs. Finally, in Fig. 15, there is shown that a three-dimensional virtual space, i.e. a three-dimensional cartesian coordinate system could potentially be used to add a further dimension to a gesture which can be used to add additional information to a played note. In various examples, any of the buttons or ancillary functions described above could be mapped to the third / depth dimension. The depth dimension could be used to change timbre / instrument or to change octave, for example. Furthermore, a plurality of interface zones (as shown in Fig. 10) could be stacked in a depth dimension so as to allow options to interact with different interface zones concurrently. Final Comments The present invention relates to the implementation of an instrument-based music control or capture system on computers and various types of mobile or assistive devices for enhanced music input and control. Specifically, the invention aims to improve the way users interact with music applications on electronic devices, providing a more intuitive, natural-feeling and efficient music input experience. The musical interface enables high level musical output from simple gesture based input on an arbitrary size grid with in two or three dimensions - including but not limited to swipe, tap, head or eye movement. Output includes but is not limited to single notes, chords, sequences of notes with varying timbral changes to the sound. The music input device is designed to empower individuals without deep music knowledge, offering an innovative solution for playing music in a user-friendly manner. Paired with the ‘Arrow Notes’ as a means or instructing or guiding the use, new users can quickly learn to play music. The system can turns any pixel array or virtual cartesian coordinate system of arbitrary size into an intuitive musical interface by mapping an interface zone to it. The invention has applications in touchscreen devices, personal computers, cloud-based solutions, AR / VR, video / mobile gaming, live events as well as studio recordings or jamming 5 sessions, allowing a musician to capture their creativity very quickly with minimal setup. A web-based or networked, classroom-based version can be used to allow tuition. A teacher / tutor could for example see live the inputs of the user and additional 10 information about to confirm how the student is playing, e.g. making it simple to spot and correct issues relating to timing, velocity or other qualities of notes being played. This information could also be provided to the individual user for selfimprovement purposes.

Claims

1. A music input system comprising:a gesture sensing device anda controller comprising machine readable instructions for generating a gesture interface zone to receive user inputs for sensing by the gesture sensing device, the controller processing sensed user gestures relative to the interface zone to generate an output signal,wherein the gesture interface zone has a plurality of regions recognised by the controller and the controller assigns different music notes or chords to user gestures within the different regions of the gesture interface zone, andthe controller identifies one or more movement parameter for the sensed user gesture, the controller appending a characteristic to the assigned music note or chord based on said movement parameter, the controller generating the output signal comprising data identifying the assigned music note or chord and the appended characteristic.

2. A data carrier comprising machine readable instructions for operation of one or more processor of a gesture sensing device as a music input system to:define a gesture interface zone for receiving user inputs;identify a plurality of different regions of the gesture interface zone and assign different music notes or chords to the different regions of the gesture interface zone;monitor one or more movement parameter for user gestures sensed within the gesture interface zone by the gesture sensing device;determine one or more characteristic for the assigned music note or chord based on said monitored movement parameter; andgenerate an output signal comprising data identifying the assigned music note or chord and the one or more characteristic appended thereto.

3. The system or data carrier of claim 1 or 2, wherein the processing of the user gesture to generate the output signal is performed onboard the gesture sensing device.

4. The system or data carrier of any preceding claim, wherein the one or more characteristic determines a quality of the assigned music note or chord that affects playback thereof.

5. The system or data carrier of any preceding claim, wherein the one or more characteristic comprises pitch and velocity for the note / chord.

6. The system or data carrier of any preceding claim, wherein the controller identifies a note on component for the output signal upon initiation of a user gesture within a region of the gesture interface zone or movement of a user gesture into a region of the gesture interface zone.

7. The system or data carrier of any preceding claim, wherein the controller identifies a note off component for the output signal upon movement of a user gesture out of a region of the gesture interface zone or termination of the gesture.

8. The system or data carrier of any preceding claim, wherein the controller determines a plurality of characteristics for the assigned music note or chord according to a further parameter of the sensed user gesture or according to a further user gesture made concurrently with the sensed user gesture.

9. The system or data carrier of any preceding claim, wherein the controller determines a plurality of characteristics for the assigned music note or chord, one or more characteristic being applied at a different time within the duration of the note / chord being played than another characteristic.

10. The system or data carrier of claim 9, wherein one or more characteristic is applied during an initial portion of a note / chord being played and a further characteristic is applied during a final portion or main portion of a note / chord being played.

11. The system or data carrier of any preceding claim, wherein the output signal comprises a Musical Instrument Digital Interface (MIDI) signal and the characteristic comprises one or more CC for the MIDI signal.

12. The system or data carrier of any preceding claim, wherein the gesture sensing device comprises a tactile interface, such as a touchscreen device.

13. The system or data carrier of any preceding claim, wherein the gesture sensing device comprises a non-contact gesture sensing device, such as an imaging device for eye-tracking, face-tracking or hand-tracking.

14. The system or data carrier of any preceding claim, wherein the gesture interface zone is two-dimensional and the regions are defined areas within the zone.

15. The system or data carrier of any preceding claim, wherein the gesture interface zone is determined reactively in response to initiation of the user input device or a user gesture.

16. The system or data carrier of any preceding claim, wherein the plurality of regions of the interface zone are adjoining, each depending radially outwardly from a central portion of the gesture interface zone to a periphery of the gesture interface zone.

17. The system or data carrier of any preceding claim, wherein the interface zone and / or plurality of regions are displayed to the user via a display.

18. The system or data carrier of any preceding claim, wherein the interface zone comprises at least four, six or eight regions of the touchscreen, each being assigned to a different musical note / chord of an octave.

19. The system or data carrier of any preceding claim, wherein the movement parameter comprises one or more of a direction, speed, acceleration, duration ordistance of a user gesture in the gesture interface zone, e.g. through / within one or more region.

20. The system or data carrier of any preceding claim, wherein a plurality of5 gesture interface zones are generated concurrently and a plurality of user gestures are input by the user or processed by the controller concurrently.

21. The system or data carrier of any preceding claim, wherein instructions for notes / chords to be played are communicated to a user as directional gesture10 inputs, e.g. in the form of arrows, an angle / orientation which denotes a region of the gesture interface zone.

22. The system or data carrier of claim 21, wherein a geometric property, shape or colour of the directional gesture inputs indicates one or more movement15 parameter to be applied as a gesture by the user.

Images