Musical instrument player hand-tracking

Abstract

The application relates to methods and systems for controlling the output of musical instruments. Information indicating positions of one or more components of a user may be correlated with musical note outputs and associated together to provide an output comprising the musical note and correlated position information.

Description

Musical Instrument player hand-trackingField

[0001] The present disclosure relates to methods and systems for controlling the output of a musical instrument. More specifically, but not exclusively, methods for creating and correlating a position of the hands of a user with the musical notes output from the musical instrument are described. In some examples, the motion of the user’s hands may further control or augment the musical output.Background

[0002] Traditionally, controlling a musical instrument requires a physical contact with keys, strings, valves, or other mechanical devices. In the case of a Theremin, a user will control the musical instrument without physical contact by acting as the grounding plate in relation to an antenna. Whilst these traditional methods of controlling musical instruments allow for a wide degree of musical expression by the user, there are gains to be made by using recent technological advancements in the area of motion tracking to control musical instruments. However, there are numerous technological challenges to be able to apply these advancements which are to at least some degree addressed or supported by the methods and systems described herein.SummaryIn a first embodiment there is provided a computer-implemented method for controlling the output of a musical instrument comprising: receiving first information indicating a first position of one or more components of a hand of a first user; detecting a musical note output from the musical instrument; correlating the first information with the musical note output to identify a first component of the one or more components of the hand of the first user, the first component associated with the musical note output; and providing a first output comprising the musical note output and the correlated first information. Sound may then be synthesised corresponding to the musical note output.The method may further comprise receiving second information indicating a second position of the one or more components of the hand of the first user; calculating a transformation between the first position and the second position to provide a transformation vector; andproviding a second output comprising the transformation vector. The first sound may be modified proportionate to the transformation vector.The method may further comprise receiving second information indicating a second position of the one or more components of the hand of the first user; and modifying the first sound in accordance with a difference between the first information and the second information.The step of receiving first information may comprise: controlling a camera to record one or more images of the hand of the first user; and processing the one or more images to extract 3D coordinates indicative of the first position of the one or more components of the hand, and wherein the first information comprises the 3D coordinates.The 3D coordinates may be indicative of the first position of the one or more components of the hand relative to a position of an interface of the musical instrument.The musical instrument may comprise a plurality of keys. For example, the musical instrument may be a piano or a keyboard. In an alternative, the musical instrument may comprise strings. In another alternative, the musical instrument may comprise drums and / or cymbals.Detecting the musical note output from the musical instrument may comprise receiving music communications protocol data from the musical instrument in the form of a note on event comprising a key identifier, and optionally wherein the music communications protocol data is in the form of MIDI data or OSC data. Correlating the first information with the musical note output may comprise: defining a bounding box area corresponding to the key identifier in 3D coordinates; comparing the bounding box area to the first position of the one or more components of the hand; and determining a most likely component of the hand out of the one or more components of the hand to be associated with the bounding box area.Comparing the bounding box area to the first position of the one or more components of the hand may comprise determining a distance between the 3D coordinates of the bounding box area and the 3D coordinates of the first position of the one or more components of the hand.Detecting the musical note output from the musical instrument may comprise receiving an audio signal from a microphone or pickup configured to detect sound emitted from the musical instrument.The step of receiving second information may comprise controlling a camera to record one or more images of the hand of the first user; and processing the one or more images to extract transformed 3D coordinates indicative of the second position of the one or more components of the hand, and wherein the second information comprises the transformed 3D coordinates. The transformed 3D coordinates may be indicative of the second position of the one or more components of the hand relative to a position of the interface of the musical instrument.Calculating a transformation between the first position and the second position may comprise subtracting the first information from the second information to provide a transformation vector, the transformation vector indicative of the degree of movement or rotation between the first position and the second position.Modifying the first sound proportionate with the transformation vector may comprise providing MIDI data of one or more of polyphonic key pressure, control change, channel pressure, and pitch wheel change. The polyphonic key pressure may comprise a key identifier and a pressure value corresponding to the transformation vector. The control change may comprise a controller number and a new value, wherein the new value corresponds to the transformation vector. The channel pressure may comprise a pressure value corresponding to the transformation vector. The pitch wheel change may comprise a new value relative to a centre value, and wherein the new value relative to the centre value corresponds to the transformation vector. The one or more components of the hand of the first user may comprise one or more of a finger-tip, knuckles, and a wrist joint of the first user. The first position may comprise translation and rotation of the one or more components of the hand of the first user.In an embodiment there is provided a synthesiser system for a musical instrument comprising a control interface, a camera directed at the control interface, and one or more processors in communication with the control interface and the camera. The one or more processors may be configured to carry out any of the methods described herein.The control interface may be a keyboard comprising a plurality of keys, and optionally each of the plurality of keys corresponds to a musical note.The camera may comprise two sensors, and optionally each of the two sensors is directed over a different portion of the control interface. The different portions of the control interface may overlap. The camera may comprise two sensors, and optionally each of the two sensors is at least partially directed over a same portion of the control interface. By providing twosensors which are at least partially directed over a same or single portion of the control interface, depth detection of objects in the field of view may be improved.In an embodiment, there is provided a method of calibrating a synthesiser system for a musical instrument comprising a control interface and a camera directed at the control interface, comprising detecting a first note of an output from the control interface corresponding to a first predefined musical note, receiving first information indicating a first position of one or more components of a hand of a first user, detecting a second note of the output from the control interface corresponding to a different, second predefined musical note, receiving second information indicating a second position of one or more components of a hand of a first user, and determining a position of the control interface relative to the camera based on the first information and the second information. Optionally, the method may comprise determining a size of the control interface, and wherein the size of the control interface may be the width of the control interface. The control interface may be a keyboard, and the first note corresponds to a first key of the keyboard and the second note corresponds to a second key of the keyboard.Optionally, the method further comprises prompting the user to press the first predefined musical note, and prompting the user to press the second predefined musical note, and wherein prompting the user to press the first and second predefined musical notes comprises displaying an indication of the first and second predefined musical notes on a screen in communication with the synthesiser or, wherein prompting the user to press the first and second predefined musical notes comprises lighting corresponding indicator lights proximate to the first and second keys. The method may further comprise defining a bounding box for each key of the keyboard, the bounding box for each key corresponding to an outer edge of each key.Brief Description of the Drawings

[0003] Exemplary arrangements of the disclosure shall now be described with reference to the drawings in which:Figure 1 illustrates a system for creating a three-dimensional model of an instrument player’s hands;Figure 2 illustrates a further system for creating a three-dimensional model of an instrument player’s hands;Figure 3 illustrates a system for controlling the output of a musical instrument; Figure 4 illustrates a system for controlling the output of a musical instrument;Figure 5 illustrates a flow chart describing a method according to the present disclosure;Figure 6 illustrates a three dimensional model of an instrument player’s hands; Figure 7 illustrates a three dimensional model of an instrument player’s hand;Figure 8 illustrates a flow chart describing a method according to the present disclosure;Figure 9 illustrates a three-dimensional model of an instrument player’s hands and a bounding box;Figure 10 illustrates a flow chart describing a method according to the present disclosure;Figure 11 illustrates a system for controlling the output of a musical instrument being calibrated.

[0004] Throughout the description and the drawings, like reference numerals refer to like parts.Specific Description

[0005] Figure 1 illustrates a system 100 for creating a three-dimensional (3D) model of a user’s hands. The system comprises at least one optical sensor, in this case a camera 102, and a keyboard 104. Whilst the embodiments described herein are shown in relation to a keyboard, it would be within the skilled person’s understanding that the methods and systems may readily be applied to other instruments including but not limited to fretted instruments (e.g. guitars, basses, banjos, ukuleles), string instruments (e.g. harps, violins, cellos, violas), brass instruments (e.g. trumpets, trombones), woodwind instruments (e.g. saxophones, clarinets, oboes), or electronic instruments (e.g. synth pads, beat pads). The keyboard 104 comprises a plurality of keys 106. When a user presses a key 106 using their hands 108 the keyboard 104 will typically cause a sound to be produced either from a speaker in the keyboard 104 itself or by connection to an external speaker. As shown in Figure 1, the camera 102 is directed such that it has a field of view 110 overlapping with at least some of the keys 106.

[0006] Figure 2 illustrates another system 100 for creating a 3D model of a user’s hands 108. The system 100 shown in Figure 2 is the same as shown in Figure 1, but includes a second camera 103. The second camera 103 may have a second field of view 111 overlapping with at least some of the keys 106. The field of view 110 of the first camera 102may be different from the field of view 111 of the second camera 103. In this way, the additional camera 103 may be used to provide information about depth of objects in view as determined by conventional stereoscopic techniques. The additional camera 103 may provide an additional field of view 111 which may compensate for obstruction by the user’s hands 108. For example, where the user’s hands cover up the view of a particular key 106 from the first camera 102, the second camera 103 may provide an unobstructed view. Additionally, or alternatively, the second camera 103 may provide a field of view 111 of keys 106 which are not covered by the field of view 110 of the first camera 102. For example, over a particularly long keyboard 104, the second camera 103 may provide coverage of additional keys 106 which are not viewable by the first camera 102. In this way, more than two cameras may be used to provide coverage of additional keys or parts of the instrument.

[0007] Figure 3 illustrates a synthesiser system 200 comprising a system 100 as illustrated in Figures 1 and 2, wherein the system 100 comprises a control interface or keyboard 104, one or more cameras 102, 103, and one or more processors 120. The one or more processors 120 are in communication with the one or more cameras 102, 103, and may optionally be in communication with the control interface 104 and a synthesiser 130. The one or more processors 120 may be configured to interpret the images received from the one or more cameras 102, 103, and the inputs received from the control interface 104. The synthesiser 130 is coupled to the control interface such that it may receive MIDI or music protocol commands from the control interface 104 including but not limited to note-on events, note-off events, polyphonic key changes, pitch wheel change, and channel pressure. The synthesiser 130 is coupled to the one or more processors 120 such that the one or more processors 120 may provide additional MIDI commands to the synthesiser 130. The additional MIDI commands may include note-on events, note-off, events, pitch wheel change, polyphonic key changes and / or channel pressure. The processor 120 may alternatively provide additional control of the synthesiser which are not provided for within the available MIDI messages. The synthesiser 130 is configured to produce a sound, or an audio signal to be provided to an amplifier based on the inputs received from the control interface 104 and the one or more processors 120. The MIDI protocol is described herein as an example of a music communications protocol which may be utilised in the scope of the present disclosure. Other music communications protocols may be readily used in place of MIDI, for example Open Sound Control (OSC) protocol, or even any other music communications protocol such as a custom music communications protocol.

[0008] The MIDI message for polyphonic key pressure comprises a key identifier (for example C3, or key 19) and a pressure value. Control of the polyphonic key pressure may be achieved by adjusting or setting the pressure value corresponding to the desired amount. The MIDI message for a control change comprises a controller number and a new value.Control of the control change may be achieved by identifying a controller (for example a pedal, or a virtual control device) and adjusting or setting the new value corresponding to the desired amount. The MIDI message for channel pressure comprises a pressure value. The channel pressure may allow for control of a currently playing musical notes by adjusting or setting the channel pressure to the desired amount. The MIDI message for a pitch wheel change comprises a value relative to a centre value. The pitch wheel change may indicate either an increase or a decrease in pitch. Therefore, by setting the value above or below the centre value, the pitch may be increased or decreased. The pitch wheel change may allow for control of the pitch upwards or downwards.

[0009] In an alternative embodiment, detecting the musical note output from the musical instrument may comprise receiving an audio signal from a microphone or pickup configured to detect sound emitted from the musical instrument. For example, in methods using an acoustic piano, or a guitar as the musical instrument, the sound emitted by the musical instrument may be detected by a microphone or other sound pickup and produce an audio signal corresponding to the detected musical note output. The sound emitted by the musical instrument may be processed similarly to any other synthesised sound by reproducing the sound at an output and varying the properties of the output sound in accordance with the user hand gesture control. As an example, a guitarist may adjust the volume or pitch of the sound output by changing the angle of the wrist. As another example, a pianist on an acoustic piano may adjust the volume or pitch of notes played on the acoustic piano by adjusting hand or finger position, allowing new ways to control the sound of acoustic instruments.

[0010] Figure 4 illustrates an alternative synthesiser system 300. Similarly to synthesiser system 200, the synthesiser system 300 comprises a system 100 as illustrated in Figures 1 and 2, one or more processors 120 and a synthesiser 130. In the synthesiser system 300 shown in Figure 4, the control interface 104 is in communication with the one or more processors 120, the one or more cameras 102, 103 are in communication with the one or more processors, and the one or more processors 120 are in communication with the synthesiser 130. However, the control interface 104 is not in direct communication with the synthesiser 130, that is, all outputs from the control interface 104 are either interpreted or relayed by the one or more processors 120. The synthesiser 130 is configured to produce a sound, or an audio signal to be provided to an amplifier based on the inputs received from the one or more processors 120.

[0011] Figure 5 illustrates a flow diagram of a method for controlling the output of a musical instrument according to an embodiment of the disclosure. The method as illustrated in Figure 5 comprises receiving first information indicating a first position of one or morecomponents of a hand of a first user. The first information may be received at the one or more processors 120 and from the one or more cameras 102,103. The one or more cameras 102, 103, may include some degree of onboard processing such that the first information received by the one or more processors 120 includes 3D geometry and identifiers indicative of components of a hand. Alternatively, the one or more cameras 102, 103 may provide images to the one or more processors 120 so that processing of the images to determine the 3D geometry and / or identifiers (e.g. fingertip, knuckle, wrist, etc) may be done by the one or more processors 120. Further alternatively, the processing of the images provided by the one or more cameras 102, 103, may be done on a separate processor not pictured, which may be located in a cloud computing environment. The one or more components of the hand of a first user may include the joints of the fingers, the thumb, and the wrist. The first position of the one or more components of the hand of the first user is the position of those components at a first point in time.

[0012] The method comprises detecting a musical note output from the musical instrument. The musical instrument may comprise the keyboard 104 as shown in Figures 1 to 4. Alternatively, the musical instrument may comprise a beat pad or other electronic input device (such as a keypad or keyboard). In further alternative embodiments, the musical instrument may comprise a fretboard or stringboard and the musical note may be detected by determining a frequency of the string note being played. In the case where the musical instrument comprises a keyboard 104, the musical note output may be in the form of a MIDI note-on event which is a digital message indicating that a note is being played and identifying the particular note which is being played.

[0013] The method comprises correlating the first information with the musical note output to identify a first component of the one or more components of the hand of the first user. The first component is associated with the musical note output. Further details of the correlation of the first information with the musical note output are provided in relation to Figure 8 and Figure 9. The method further comprises providing a first output comprising the musical note output and the correlated first information. The musical note output may be straightforward to determine since the MIDI message from the keyboard 104 identifies the particular key or musical note which is to be played. However, it is by correlating that musical note output with the identified first component of the one or more components of the hand of the first user that the method provides additional data which may be used for guiding the user interaction or for the user to further control the output of the musical instrument. For example, where a scale is being played on a keyboard 104, the provision of the correlated first information may identify that a note is being played by the first finger of the user.Subsequent notes may be identified as correlating with the second and third fingers of the user. At an appropriate point in the scale, the user may be required to shift the hand so thatfurther keys may be correctly played in succession. The first output may allow for the user to be assessed on correct fingering or guided by feedback from the musical instrument or synthesiser system to correct the fingering where the wrong finger has been used to press a particular key. Further, the musical instrument may be configured to respond differently depending on the particular component of the hand which is being used. For example, a drum pad or keyboard may produce one particular sound when fingers from one hand are used to play the notes, but produce a different sound when fingers from the other hand are used to play the same notes.

[0014] Optionally, the method may comprise synthesising a first sound corresponding to the musical note output. The sound may be a musical note generated or recorded and played back which corresponds to the particular key which has been pressed. The first sound may be produced from a speaker locally, or the first sound may be passed to another amplifier which can produce the sound. For example, the sound may be emitted from a speaker on the musical instrument, or it may be produced on a connected device (such as a tablet, a computer, a smartphone, or wireless speaker).

[0015] In a further step of the method according to an embodiment, the method may comprise receiving second information indicating a second position of the one or more components of the hand of the first user. The second position of the one or more components of the hand of the first user may correspond to a position in which the user has adjusted the position of their hand, wrist, or fingers, such that the second position is different to the first position. As a non-limiting example, the user may press a key whilst the hand is in the first position and, whilst continuing to hold the key, the user may straighten or bend the finger holding the key around one of the knuckles. The second position of the one or more components of the hand will correspond to the straightened or bent position of the finger holding the key.

[0016] In a further step of the method according to an embodiment, the method may comprise calculating a transformation between the first position and the second position to provide a transformation vector. Figure 6 illustrates a non-limiting example of one or more components of the hands of a user which may be identified to correlate with the musical note output. As shown in Figure 6, there are points at each of the wrists, the finger tips, and each knuckle of the figures within the field of view. Points corresponding to components of the hand which are outside of the field of view may be implied by other known information such as the length of the fingers, or last known position. Each point which represents a component of the user’s hands may be provided in the form of a vector having six entries. As shown in Figure 7, the six entries of the vector for each point may correspond to three orthogonal (x, y, z) position components and three corresponding angular rotation components (0X, 0y, 0Z). When the hand has moved to the second position and the components of the hand are indifferent orthogonal positions and angular rotations, each component may comprise a new updated vector comprising three orthogonal position components and three angular rotation components. For each component, the difference between the first vector and the second updated vector may be taken to calculate the transformation between the first position and the second position. The transformation may be provided in the form of a third, transformation vector. For example, if the rotation of a finger joint is desired to control the output of the musical instrument by bending the finger to one side, the transformation vector 0yvalue for one knuckle of the finger may be determined and passed to the next step of the method to modify the synthesised sound.

[0017] The method may include modifying the synthesised sound in accordance with the transformation. Continuing the example given above, wherein the rotation of the knuckle of a finger is being used to modify the sound, the 0yvalue of the transformation vector may be used to modify the sound. Modification of the sound may be done by providing an additional MIDI control signal such as changing the pitch of the musical note, the aftertouch of the musical note, or it may be done by directly controlling the output without the use of MIDI messages to, for example, control the volume of the note output. Whilst the example of rotating a knuckle of a finger to modify the sound has been given, the methods may be applied to a number of different configurations within the scope of the claims. For example, the wrist position or rotation may be used to modify the sound of one or more notes. Alternatively, the change in position of a different hand of the user may be used to modify the sound. By associating the musical note with the correct finger playing the note, the method allows for each individual finger to provide control or modification of the sounds produced without affecting other notes being played. For example, by straightening the first finger playing a first note, the volume or pitch of the first note may be modified without modifying the volume or pitch of other notes being played simultaneously.

[0018] Figure 8 illustrates further details of an exemplary method for receiving the first information indicating the positions of one or more components of the user’s hand. The step of receiving the first information as shown in Figure 5 may further comprise controlling a camera to record one or more images of the hand of the first user. The one or more processors may be configured to communicate with the one or more cameras such that they can control the triggering of the cameras to provide images of the hands of a user. The one or more cameras may also be controlled to adjust camera settings such as exposure time, capture frame rate, gain, and white balance. The method may further comprise processing the one or more images to extract 3D coordinates indicative of the first position of the one or more components of the hand. The first information may comprise those 3D coordinates. As shown in Figure 7, the one or more processors may process the images of the hand to determine locations of one or more components of the hand. Conventional techniques foridentifying components of the hand may be used including Al based methods and feature detection mapping. The output from processing the images of the hand may be sets of 3D coordinates of the features of the hand. For example, the output from processing the images may include a plurality of vectors containing the 3D position and angular rotation as 3D coordinates. Each of the vectors may be appropriately labelled to correspond to a particular feature of the hand, for example, the wrist, the first finger tip, the first finger first knuckle, the first finger second knuckle, the first finger third knuckle, etc. The 3D coordinates may be used as described herein to provide the data regarding the first position and the second position of the one or more components of the hand and to calculate the transformation vectors. In particular, the 3D coordinates of a component of the hand in the first position may be subtracted from the 3D coordinates of the component of the hand in the second position to provide the transformation vector. One or more components of that transformation vector may be used to modify the synthesising of the sound as described herein.

[0019] Similarly to the step of receiving first information described herein, the step of receiving second information may also comprise controlling the one or more cameras to record one or more images of the hand of the first user, and processing the one or more images to extract second 3D coordinates which correspond to a transformed position of the hand. The transformed 3D coordinates may be in the same or similar form to the 3D coordinates of the first information such that together they may be combined to calculate the transformation vector for the transformation between the first position and the second position of the user’s hands.

[0020] Figure 9 illustrates a keyboard being played by a user wherein the 3D coordinates of the one or more features of the hands have been overlaid on the illustration. In addition, overlaid on the illustration is a depiction of a bounding box area 910 as may be used in some methods according to a further embodiment in which correlating the first information with the musical note is performed. As illustrated in Figure 10, the step of correlating the first information with the musical note output may comprise defining a bounding box area 910 corresponding to the key identifier in 3D coordinates, comparing the bounding box area 910 to the first position of the one or more components of the hand, and determining a most likely component of the hand out of the one or more components of the hand to be associated with the bounding box area 910. A representation of a bounding box area 910 is shown in Figure 9 in broken lines. The bounding box area 910 may be defined as a volume in 3D space or as a discrete surface which corresponds to a key 106 of the keyboard 104. The bounding box area 910 may be defined to cover the entirety of the playing surface of a single key, or a volume in which a finger may be located when it is interacting with the key 106. The bounding box area 910 may be defined by one or more 3D coordinates corresponding to vertices of the key 106, or by any suitable geometry constructto define the volume or playing surface. In an example, as shown in Figure 9, a key has been pressed. The bounding box area of that key is known and is illustrated in Figure 9. The 3D coordinates of each of the fingertips of each of the user’s hands is known from the steps of the method extracting those 3D coordinates from the one or more images of the hands. In a further step of the method, the 3D coordinates of each of the finger tips may be compared to the 3D coordinates of the bounding box area to determine whether one of the fingertips is located within the bounding box area. Since the 3D coordinates may not be precisely aligned with the bounding box area, the method may include determining a likelihood for each component of the user’s hands that that component was the component of the hand associated with the key press by its proximity to the bounding box area. Where a most likely candidate component is identified, the identity of this component may be provided in combination with the note-on command for further processing, analysis, or control of the musical instrument.

[0021] It will be appreciated that the 3D coordinates for each of the components of the hand, and for the bounding box area may have a reference point in space corresponding to a zero position and zero angular rotation. Such a reference position may be defined in relation to the keys of the keyboard, the playing surface of a musical instrument, or it may be redefined in respect of the first information. For example, when the first position is defined, that position may be taken to be the zero reference point. In relation to that reference point, the second position may be determined and the location of the bounding box area may be determined. Alternatively, the zero reference point may be defined within the bounding box area in response to a note-on MIDI event. Further alternatively, the method may simply use a camera centric zero reference which is maintained in position relative to the one or more cameras.

[0022] Figure 11 illustrates part of a synthesiser system for a musical instrument being used to calibrate the systems and methods as described herein. In a first calibration step the method of calibrating comprises detecting a first note of an output from the control interface corresponding to a first predefined musical note. The musical note is predefined in that the musical note is a known musical note as part of the method. The musical note may be known because it has been predefined and the user has been prompted to press the predefined musical note. In the example shown in Figure 11, the user is prompted to press the highlighted key on the screen. Alternatively, the user may be given a musical note identifier (such as C3, or middle C) to press. Additionally or alternatively, the control interface may be configured to highlight the predefined musical note by illuminating an LED corresponding to that musical note. In the example shown, the LED is contained within a partially transparent or translucent key such that illuminating the LED illuminates the corresponding key of the keyboard.

[0023] The method of calibrating further comprises receiving first information indicating a first position of one or more components of a hand of a first user. Receiving the first information may be carried out according to the other methods described herein.

[0024] In a further step of the method of calibrating, the method comprises detecting a second note of the output from the control interface corresponding to a different, second predefined musical note. The first and second musical notes may be relatively close to each other on the control surface (e.g. keys in the same octave) or they may be located at far ends of the control surface (e.g. at either end of the control surface). The method further comprises receiving second information indicating a second position of one or more components of a hand of a first user. Receiving the second information may be carried out according to the other methods described herein in relation to receiving first information and second information.

[0025] The method further comprises determining a position of the control interface relative to the camera based on the first information and the second information. Since the two predefined musical notes are known, the location of the users hands or fingers when activating these musical notes may be used to imply a position of the control surface. The method may be extended to more than two predefined musical notes such that an array of musical notes may be located according to the 3D coordinates of the components of the hands when activating the musical notes or pressing the keys of the keyboard. The location of the control surface may be used to provide a zero reference point as used in other methods described herein. The location of the control surface may provide information to allow for the determination of bounding box areas for each of the musical notes or keys.

[0026] Optionally, the method may comprise determining a size of the control interface. In some examples, keyboards may comprise different numbers of keys and those keys may be of varying widths (for example a full size piano may have wider keys and more keys than a portable keyboard or synthesiser). By determining the size, or by determining the width of the control interface, the bounding box areas may be appropriately distributed across the field of view of the one or more cameras without requiring the end keys of the control surface to be pressed.

[0027] Methods and systems have been described herein with reference to the MIDI protocol. It will be understood that the scope of the present disclosure encompasses any other music communications protocols including but not limited to OSC, MIDI Polyphonic Expression or custom music communications protocols. It will be understood that synthesis of sound by the synthesiser or in any other system described herein may include generating a new musical sound, reproducing a recorded sound, amplifyying a live recorded sound. The systems and methods described herein can be used with any conventional musical note synthesis.

[0028] While the methods and systems described herein describe an implementation including one or more cameras, it will be appreciated that the systems and methods may be applied using any number of cameras or any number of sensors having optical or physical measurement of a user within a field of view or sensory field. For example, optical measurement sensors such as laser based optical sensors, tracking motion sensors relying on structured light or time of flight calculations, 3D scanners, or physical sensors mounted on the hand or a glove may be used to determine the motion and / or position of the hands of a user.

[0029] The methods and systems described herein have been described with reference to vectors encoded by reference to three position coordinates and three rotation coordinates. It will be appreciated that any suitable representation of the hand of a user may be used to encode position or motion of the hand without departing from the methods and systems described herein. For example, positions and / or motions of the hands of a user may be encoded in quaternion notation to represent rotation about an origin within three-dimensional Euclidean space. Representation of the position and / or motion of the hands of a user may further be constrained based on inverse kinematics. That is, the position and or motion of one element may be constrained with respect to another element. For example, the position of the tip of a finger may simply be encoded as the angle with respect to a joint of the finger, rather than using the full unambiguous coordinates for the tip of the finger.

Claims

Claims1. A computer-implemented method for controlling the output of a musical instrument comprising: receiving first information indicating a first position of one or more components of a hand of a first user; detecting a musical note output from the musical instrument; correlating the first information with the musical note output to identify a first component of the one or more components of the hand of the first user, the first component associated with the musical note output; and providing a first output comprising the musical note output and the correlated first information.

2. The computer-implemented method of claim 1 , further comprising synthesising a first sound corresponding to the musical note output.

3. The computer-implemented method of claim 1 or claim 2, further comprising receiving second information indicating a second position of the one or more components of the hand of the first user; calculating a transformation between the first position and the second position to provide a transformation vector; and providing a second output comprising the transformation vector.

4. The computer-implemented method of claim 3 when dependent on claim 2, further comprising modifying the first sound proportionate with the transformation vector.

5. The computer-implemented method of any preceding claim, wherein the step of receiving first information comprises: controlling a camera to record one or more images of the hand of the first user; and processing the one or more images to extract 3D coordinates indicative of the first position of the one or more components of the hand, and wherein the first information comprises the 3D coordinates.

6. The computer-implemented method of claim 5, wherein the 3D coordinates are indicative of the first position of the one or more components of the hand relative to a position of an interface of the musical instrument.

7. The computer-implemented method of any preceding claim, wherein the musical instrument comprises a plurality of keys.

8. The computer-implemented method of claim 7, wherein detecting the musical note output from the musical instrument comprises receiving music communications protocol data from the musical instrument in the form of a note on event comprising a key identifier, and optionally wherein the music communications protocol data is in the form of MIDI data or OSC data.

9. The computer-implemented method of claim 8, wherein correlating the first information with the musical note output comprises: defining a bounding box area corresponding to the key identifier in 3D coordinates; comparing the bounding box area to the first position of the one or more components of the hand; and determining a most likely component of the hand out of the one or more components of the hand to be associated with the bounding box area.

10. The computer-implemented method of claim 9, wherein comparing the bounding box area to the first position of the one or more components of the hand comprises determining a distance between the 3D coordinates of the bounding box area and the 3D coordinates of the first position of the one or more components of the hand.

11. The computer-implemented method of any of claims 1 to 6, wherein detecting the musical note output from the musical instrument comprises receiving an audio signal from a microphone or pickup configured to detect sound emitted from the musical instrument.

12. The computer-implemented method of claim 3, or any of claims 4 to 11 when dependent on claim 3, wherein the step of receiving second information comprises:controlling a camera to record one or more images of the hand of the first user; and processing the one or more images to extract transformed 3D coordinates indicative of the second position of the one or more components of the hand, and wherein the second information comprises the transformed 3D coordinates.

13. The computer-implemented method of claim 12, wherein the transformed 3D coordinates are indicative of the second position of the one or more components of the hand relative to a position of the interface of the musical instrument.

14. The computer-implemented method of claim 3, or any of claims 4 to 13 when dependent on claim 3, wherein calculating a transformation between the first position and the second position comprises subtracting the first information from the second information to provide a transformation vector, the transformation vector indicative of the degree of movement or rotation between the first position and the second position.

15. The computer-implemented method of claim 4, or any of claims 5 to 14 when dependent on claim 4, wherein modifying the first sound proportionate with the transformation vector comprises providing MIDI data of one or more of polyphonic key pressure, control change, channel pressure, and pitch wheel change.

16. The computer-implemented method of claim 15, wherein the polyphonic key pressure comprises a key identifier and a pressure value corresponding to the transformation vector.

17. The computer-implemented method of claim 15, wherein the control change comprises a controller number and a new value, wherein the new value corresponds to the transformation vector.

18. The computer-implemented method of claim 15, wherein the channel pressure comprises a pressure value corresponding to the transformation vector.

19. The computer-implemented method of claim 15, wherein the pitch wheel change comprises a new value relative to a centre value, and wherein the new value relative to the centre value corresponds to the transformation vector.

20. The computer-implemented method of any preceding claim, wherein the one or more components of the hand of the first user comprise one or more of a fingertip, knuckles, and a wrist joint of the first user.

21. The computer-implemented method of any preceding claim wherein the first position comprises translation and rotation of the one or more components of the hand of the first user.

22. A synthesiser system for a musical instrument comprising: a control interface; a camera directed at the control interface; and one or more processors in communication with the control interface and the camera, wherein the one or more processors are configured to carry out the method according to any of claims 1 to 21.

23. The synthesiser system according to claim 22, wherein the control interface is a keyboard comprising a plurality of keys, and optionally wherein each of the plurality of keys corresponds to a musical note.

24. The synthesiser system according to claim 22 or claim 23, wherein the camera comprises two sensors, and optionally wherein each of the two sensors is directed over a different portion of the control interface, and further optionally wherein the different portions of the control interface overlap.

25. A method of calibrating a synthesiser system for a musical instrument comprising a control interface and a camera directed at the control interface, comprising: detecting a first note of an output from the control interface corresponding to a first predefined musical note; receiving first information indicating a first position of one or more components of a hand of a first user; detecting a second note of the output from the control interface corresponding to a different, second predefined musical note; receiving second information indicating a second position of one or more components of a hand of a first user; and determining a position of the control interface relative to the camera based on the first information and the second information.

26. The method of claim 25, further comprisingdetermining a size of the control interface, optionally wherein the size of the control interface is the width of the control interface.

27. The method of claim 25 or claim 26, wherein the control interface is a keyboard, and wherein the first note corresponds to a first key of the keyboard and the second note corresponds to a second key of the keyboard.

28. The method of any of claims 25 to 27, further comprising: prompting the user to press the first predefined musical note; and prompting the user to press the second predefined musical note; and wherein prompting the user to press the first and second predefined musical notes comprises displaying an indication of the first and second predefined musical notes on a screen in communication with the synthesiser or, wherein prompting the user to press the first and second predefined musical notes comprises lighting corresponding indicator lights proximate to the first and second keys.

29. The method of any of claims 25 to 28, further comprising defining a bounding box for each key of the keyboard, the bounding box for each key corresponding to an outer edge of each key.

Images