Tracking the hands of a musical instrument player

Abstract

This application relates to a method and system for controlling the output of a musical instrument. Information indicating the position of one or more user components can be correlated with a note output and associated to provide an output together that includes the note and the correlated position information.

Description

Technical Field

[0001] The present disclosure relates to methods and systems for controlling the output of musical instruments. More specifically, but not limited thereto, a method for creating the position of a user's hand and correlating it with the notes output from a musical instrument is described. In some examples, the movement of the user's hand may further control or expand the music output.

Background Art

[0002] Conventionally, controlling a musical instrument has required physical contact with a keyboard, strings, valves, or other mechanical devices. In the case of a theremin, the user controls the instrument without physical contact by functioning as a ground plane for the antenna. These conventional methods of musical instrument control enable a wide range of musical expressions by the user, but there are benefits to be gained by controlling musical instruments using recent technological advancements in the field of motion tracking. However, there are a number of technical challenges in making these advancements applicable, and these are addressed or at least somewhat accommodated by the methods and systems described herein.

Summary of the Invention

[0003] In a first embodiment, a computer-implemented method for controlling the output of a musical instrument, comprising receiving first information indicating a first position of one or more components of a first user's hand, detecting a note output from the musical instrument, correlating the first information and the note output to identify a first component of the one or more components of the first user's hand associated with the note output, and providing a first output comprising the note output and the correlated first information. Thereafter, a sound may be synthesized corresponding to the note output.

[0004] The method may further comprise receiving second information indicating the second position of one or more components of a first user's hand, calculating a transformation between the first and second positions to provide a transformation vector, and providing a second output having the transformation vector. The first sound may be modified in proportion to the transformation vector.

[0005] This method may further include receiving second information indicating a second position of one or more components of a first user's hand, and modifying a first sound according to the difference between the first and second positions.

[0006] The step of receiving first information may include controlling a camera to record one or more images of a first user's hand, and processing one or more images to extract 3D coordinates indicating a first position of one or more components of the hand, wherein the first information comprises 3D coordinates.

[0007] The 3D coordinates may indicate the first position of one or more components of the hand relative to the position of the instrument's interface.

[0008] An instrument may have multiple keys. For example, an instrument may be a piano or a keyboard. Alternatively, an instrument may have strings. In yet another alternative, an instrument may have drums and / or cymbals.

[0009] Detecting note output from an instrument may involve receiving music communication protocol data from the instrument in the form of a note-on event with a key identifier, and optionally the music communication protocol data may be in the form of MIDI data or OSC data. Correlating the first information with the score output may involve defining a bounding box region corresponding to the key identifier in 3D coordinates, comparing the bounding box region with a first position of one or more components of the hand, and determining which of the one or more components of the hand is most likely to be associated with the bounding box region.

[0010] Comparing the bounding box region with the first position of one or more components of the hand may involve determining the distance between the 3D coordinates of the bounding box region and the 3D coordinates of the first position of one or more components of the hand.

[0011] Detecting note output from an instrument may involve receiving an audio signal from a microphone or pickup configured to detect sound emitted from the instrument.

[0012] The step of receiving second information comprises controlling a camera to record one or more images of a first user's hand and processing one or more images to extract transformed 3D coordinates indicating a second position of one or more components of the hand, wherein the second information comprises transformed 3D coordinates. The transformed 3D coordinates may indicate a second position of one or more components of the hand relative to the position of the instrument interface.

[0013] Calculating the transformation between a first position and a second position may involve subtracting the first information from the second information in order to provide a transformation vector indicating the degree of movement or rotation between the first and second positions.

[0014] Modifying the first tone in proportion to the transformation vector may involve providing one or more MIDI data of polyphonic key pressure, control change, channel pressure, and pitch wheel change. Polyphonic key pressure may include a key identifier and pressure value corresponding to the transformation vector. Control change may include a controller number and a new value, the new value corresponding to the transformation vector. Channel pressure may include a pressure value corresponding to the transformation vector. Pitch wheel change may include a new value relative to the center value, the new value relative to the center value corresponding to the transformation vector. One or more components of the first user's hand may include one or more of the first user's fingertips, knuckles, and wrist joints. The first position may include translation and rotation of one or more components of the first user's hand.

[0015] In one embodiment, a musical instrument synthesizer system is provided, comprising a control interface, a camera directed to the control interface, and one or more processors communicating with the control interface and the camera. The one or more processors may be configured to perform any of the methods described herein.

[0016] The control interface may be a keyboard with multiple keys, each of which may optionally correspond to a musical note.

[0017] The camera may have two sensors, each optionally directed towards a different part of the control interface. These different parts of the control interface may overlap. The camera may have two sensors, each optionally directed at at least partially the same part of the control interface. By providing two sensors directed at at least partially the same or a single part of the control interface, depth detection of objects in the field of view may be improved.

[0018] In one embodiment, a method is provided for calibrating a musical instrument synthesizer system comprising a control interface and a camera directed toward the control interface, comprising: detecting a first note from the output of a control interface corresponding to a first predefined note; receiving first information indicating the first position of one or more components of a first user's hand; detecting a second note from the output of a control interface corresponding to a different second predefined note; receiving second information indicating the second position of one or more components of a first user's hand; and determining the position of the control interface relative to the camera based on the first and second information. Optionally, the method may also comprise determining the size of the control interface, which may be the width of the control interface. The control interface may be a keyboard, where the first note corresponds to a first key on the keyboard, and the second note corresponds to a second key on the keyboard.

[0019] Optionally, this method further comprises prompting the user to press a first predefined musical note and prompting the user to press a second predefined musical note, and prompting the user to press the first and second predefined musical notes comprises displaying an indication of the first and second predefined musical notes on a screen communicating with the synthesizer, or prompting the user to press the first and second predefined musical notes comprises lighting corresponding indicator lights near the first and second keys. This method may further comprise defining a bounding box for each key of the keyboard, and the bounding box for each key corresponds to the outer edge of each key.

[0020] A typical configuration of the present disclosure will now be described with reference to the drawings.

Brief Description of the Drawings

[0021] [Figure 1] Illustrates a system for creating a three-dimensional model of a musical instrument player's hand. [Figure 2] Illustrates a further system for creating a three-dimensional model of a musical instrument player's hand. [Figure 3] Illustrates a system for controlling the output of a musical instrument. [Figure 4] Illustrates a system for controlling the output of a musical instrument. [Figure 5] Illustrates a flowchart for explaining the method according to the present disclosure. [Figure 6] Illustrates a three-dimensional model of a musical instrument player's hand. [Figure 7] Illustrates a three-dimensional model of a musical instrument player's hand. [Figure 8] Illustrates a flowchart for explaining the method according to the present disclosure. [Figure 9] Illustrates a three-dimensional model of a musical instrument player's hand and a bounding box. [Figure 10] Illustrates a flowchart for explaining the method according to the present disclosure. [Figure 11] Illustrate a system for controlling the output of an instrument during calibration.

Best Mode for Carrying Out the Invention

[0022] Throughout the description and the drawings, like reference numerals refer to like parts.

[0023] FIG. 1 illustrates a system 100 for creating a three-dimensional (3D) model of a user's hand. This system includes at least one optical sensor, in this case a camera 102, and a keyboard 104. Although the embodiments described herein are shown with respect to a keyboard, it is within the understanding of those skilled in the art that this method and system can be readily applied to other musical instruments including, but not limited to, stringed instruments with frets (e.g., guitars, basses, banjos, ukuleles), string instruments (e.g., harps, violins, cellos, violas), brass instruments (e.g., trumpets, trombones), woodwind instruments (e.g., saxophones, clarinets, oboes), or electronic musical instruments (e.g., synthesizer pads, beat pads). The keyboard 104 includes a plurality of keys 106. When the user presses a key 106 with hand 108, the keyboard typically generates sound from a speaker of the keyboard 104 itself or through connection to an external speaker. As shown in FIG. 1, the camera 102 is oriented to have a field of view 110 that at least partially overlaps some of the keys 106.

[0024] Figure 2 illustrates another system 100 for creating a 3D model of a user's hand 108. The system 100 shown in Figure 2 is the same as the one shown in Figure 1, but includes a second camera 103. The second camera 103 may have a second field of view 111 that overlaps with at least some keys 106. The field of view 110 of the first camera 102 may be different from the field of view 111 of the second camera 103. Thus, the additional camera 103 may be used to provide information about the depth of objects in the field of view, as determined by conventional stereoscopic techniques. The additional camera 103 may provide an additional field of view 111 that can compensate for occlusion by the user's hand 108. For example, if the user's hand obstructs the view from the first camera 102 to several keys 106, the second camera 103 may provide an unobstructed view. As an addition or alternative, a second camera 103 may provide a field of view 111 of keys 106 that are not covered by the field of view 110 of the first camera 102. For example, particularly over a long keyboard 104, the second camera 103 may provide coverage of additional keys 106 that are not visible to the first camera 102. In this way, two or more cameras may provide coverage of additional keys or instrument parts.

[0025] Figure 3 shows a synthesizer system 200 comprising system 100 as illustrated in Figures 1 and 2, system 100 comprising a keyboard 104, one or more cameras 102, 103, and one or more processors 120. One or more processors 120 communicate with one or more cameras 102, 103 and optionally communicate with a control interface 104 and a synthesizer 130. One or more processors 120 may be configured to interpret images received from one or more cameras 102, 103 and inputs received from the control interface 104. The synthesizer 130 is coupled to the control interface so that it can receive MIDI or musical protocol commands from the control interface 104, including but not limited to note-on events, note-off events, polyphonic key changes, pitch wheel changes, and channel pressure. The synthesizer 130 is coupled to one or more processors 120 so that one or more processors 120 can provide the synthesizer 130 with additional MIDI commands. Additional MIDI commands may include note-on events, note-off events, pitch wheel changes, polyphonic key changes, and / or channel pressure. Alternatively, processor 120 may provide additional controls for the synthesizer not provided within the available MIDI messages. Synthesizer 130 is configured to generate audio signals to be supplied to a sound or amplifier based on inputs received from control interface 104 and one or more processors 120. The MIDI protocol is described herein as an example of a musical communication protocol that may be used within the scope of this disclosure. Instead of MIDI, other musical communication protocols, such as the Open Sound Control (OSC) protocol, or any other musical communication protocol, such as a custom musical communication protocol, may readily be used.

[0026] MIDI messages for polyphonic key pressure include a key identifier (e.g., C3 or key 19) and a pressure value. Control of polyphonic key pressure can be achieved by adjusting or setting the pressure value to a desired amount. MIDI messages for control change include a controller number and a new value. Control of a control change can be achieved by identifying the controller (e.g., a pedal or virtual control device) and adjusting or setting the new value to a desired amount. MIDI messages for channel pressure include a pressure value. Channel pressure can enable control of the currently played note by adjusting or setting the channel pressure to a desired amount. MIDI messages for pitch wheel change include a value relative to the center value. Pitch wheel change can indicate an increase or decrease in pitch. Therefore, the pitch can increase or decrease by setting a value above or below the center value. Pitch wheel change can enable control of pitch up or down.

[0027] In an alternative embodiment, detecting a note output from an instrument may involve receiving an audio signal from a microphone or pickup configured to detect sounds emitted from the instrument. For example, in a method using an acoustic piano or guitar as an instrument, the sounds emitted by the instrument may be detected by a microphone or other acoustic pickup, which may generate an audio signal corresponding to the detected note output. The sounds emitted by the instrument may be processed in the same way as any other synthetic sound by reproducing the sound at the output and changing the characteristics of the output sound according to the user's hand gesture control. For example, a guitarist may adjust the volume or pitch of the output sound by changing the angle of their wrist. Another example is an acoustic piano player who may adjust the volume or pitch of the notes played on the acoustic piano by adjusting the position of their hands and fingers, enabling new ways of controlling the sound of an acoustic instrument.

[0028] Figure 4 illustrates an alternative synthesizer system 300. Similar to synthesizer system 200, synthesizer system 300 comprises a system 100, one or more processors 120, and a synthesizer 130, as illustrated in Figures 1 and 2. In the synthesizer system 300 shown in Figure 4, a control interface 104 communicates with one or more processors 120, one or more cameras 102, 103 communicate with one or more processors, and one or more processors 120 communicate with the synthesizer 130. However, the control interface 104 does not communicate directly with the synthesizer 130; that is, all outputs from the control interface 104 are interpreted or reproduced by one or more processors 120. The synthesizer 130 is configured to generate an audio signal to be supplied to a sound or amplifier based on inputs received from one or more processors 120.

[0029] Figure 5 illustrates a flowchart of a method for controlling the output of a musical instrument according to an embodiment of the present disclosure. The method illustrated in Figure 5 comprises receiving first information indicating the first position of one or more components of a first user's hand. The first information may be received by one or more processors 120 from one or more cameras 102, 103. One or more cameras 102, 103 may include some onboard processing such that the first information received by one or more processors 120 includes 3D geometry and identifiers indicating components of the hand. Alternatively, one or more cameras 102, 103 may provide images to one or more processors 120 so that image processing for determining 3D geometry and / or identifiers (e.g., fingertips, knuckles, wrists, etc.) may be performed by one or more processors 120. Furthermore, the processing of images provided by one or more cameras 102, 103 may be performed by another processor (not shown) which may be located in a cloud computing environment. One or more components of a first user's hand may include knuckles, thumbs, and wrists. The first position of one or more components of the first user's hand is the position of those components at a first point in time.

[0030] This method comprises detecting note output from an instrument. The instrument may include a keyboard 104 as shown in Figures 1 to 4. Alternatively, the instrument may include a beat pad or other electronic input device (e.g., a keypad or keyboard). In a further alternative embodiment, the instrument may include a fretboard or stringed instrument, and the note may be detected by determining the frequency of the string being played. If the instrument includes a keyboard 104, the note output may be in the form of a MIDI note-on event, which is a digital message indicating that a note is being played and identifying the specific note being played.

[0031] This method comprises correlating first information and note outputs to identify a first component of one or more components of a first user's hand. The first component is associated with a note output. Further details of the correlation between the first information and note outputs are provided in reference to Figures 8 and 9. This method further comprises providing a first output comprising a note output and the correlated first information. The note output can be easily determined to identify a specific key or note on which a MIDI message from keyboard 104 is played. However, by correlating the note output with an identified first component of one or more components of a first user's hand, this method provides additional data that can be used to guide user interaction or to allow the user to further control the output of the instrument. For example, if a scale is being played on keyboard 104, providing the correlated first information identifies that the note is being played by the user's first finger. Subsequent notes may be identified as correlations with the user's second and third fingers. At appropriate points in the scale, the user may need to switch hands so that further keys can be played precisely in succession. The first output allows the user to be evaluated on the accuracy of their fingering, or to be guided by feedback from the instrument or synthesizer system to correct their fingering if the wrong finger is used to press a particular key. Furthermore, the instrument may be configured to respond differently depending on the specific components of the hand being used. For example, a drum pad or keyboard may produce a certain sound when playing a note using the fingers of one hand, but produce a different sound when playing the same note using the fingers of the other hand.

[0032] Optionally, this method may include synthesizing a first sound corresponding to a note output. This sound may be a note corresponding to a specific key pressed, which is generated or recorded and played back. The first sound may be generated locally from a speaker, or the first sound may be passed to another amplifier capable of generating sound. For example, the sound may be emitted from a speaker on an instrument, or it may be generated on a connected device (e.g., a tablet, computer, smartphone, or wireless speaker).

[0033] In a further step of the method according to the embodiment, the method may include receiving second information indicating a second position of one or more components of a first user's hand. The second position of one or more components of the first user's hand may correspond to a position in which the user has adjusted the position of their hand, wrist, or fingers such that the second position is different from the first position. In a non-limiting example, the user may press a key with their hand in the first position and, while continuing to press the key, extend or bend the finger pressing the key around one of its joints. The second position of one or more components of the hand may correspond to the extended or bent position of the finger pressing the key.

[0034] In a further step of the method according to the embodiment, the method may comprise calculating a transformation between a first position and a second position to provide a transformation vector. Figure 6 illustrates a non-limiting example of one or more components of a user's hand that can be identified for correlation with a musical note output. As shown in Figure 6, there are points in the field of view of the figure at the wrist, fingertips, and each joint. Points corresponding to hand components outside the field of view may be implied by other known information, such as finger length, last known position, etc. Each point representing a component of the user's hand may be provided in the form of a vector having six elements. As shown in Figure 7, the six elements of the vector for each point are three orthogonal (x,y,z) position components and three corresponding angular rotation components (θ x ,θ y ,θ z) may correspond to ). When the hand moves to a second position and the components of the hand are in different orthogonal positions and angular rotations, each component may have a new updated vector having three orthogonal position components and three angular rotation components. For each component, the difference between the first vector and the second updated vector may be taken to calculate the transformation between the first and second positions. The transformation may be provided in the form of a third transformation vector. For example, if rotation of the finger joints is desired to control the output of the instrument by bending the fingers to one side, a transformation vector θ with respect to one joint of the finger may be provided. y This is determined and can then be passed on to the next step in the method for correcting the synthesized sound.

[0035] This method may include modifying the synthesized sound according to the transformation. Continuing the above example where rotation of the finger joints is used to modify the sound, the θ of the transformation vector y Values ​​may be used to modify the sound. Sound modification may be performed by providing additional MIDI control signals, such as changing the pitch of a note or aftertouching a note, or by directly controlling the output without using MIDI messages to control the volume of the note output, for example. While an example of rotating a finger joint to modify a sound has been given, this method may be applied to numerous different configurations within the claims. For example, wrist position or rotation may be used to modify one or more notes. Alternatively, a change in the position of the user's other hand may be used to modify a sound. By associating a note with the correct finger playing that note, this method makes it possible for individual fingers to provide control or modification of the sound produced without affecting other notes being played. For example, by extending the first finger playing the first note, the volume or pitch of the first note may be modified without changing the volume or pitch of other notes being played simultaneously.

[0036] Figure 8 illustrates further details of a typical method for receiving first information indicating the position of one or more components of a user's hand. The step of receiving first information, as shown in Figure 5, may further comprise controlling a camera to record one or more images of a first user's hand. One or more processors may be configured to communicate with one or more cameras so that they can control the triggers of the cameras to provide images of the user's hand. One or more cameras may also be controlled to adjust camera settings, such as exposure time, capture frame rate, gain, and white balance. This method may further comprise processing one or more images to extract 3D coordinates indicating the first positions of one or more components of the hand. The first information may comprise their 3D coordinates. As shown in Figure 7, one or more processors may process the hand image to determine the positions of one or more components of the hand. Conventional techniques for identifying hand components, including AI-based methods and feature detection mapping, may be used. The output of processing the hand image may be a set of 3D coordinates of hand features. For example, the output of processing the image may comprise a plurality of vectors, with 3D position and angular rotation as 3D coordinates. Each vector may be appropriately labeled to correspond to a specific feature of the hand, such as the wrist, the first fingertip, the first joint of the first finger, the second joint of the first finger, the third joint of the first finger, etc. 3D coordinates may be used to calculate a transformation vector, providing data on the first and second positions of one or more components of the hand, as described herein. In particular, to provide a transformation vector, the 3D coordinates of the hand components at the first position may be subtracted from the 3D coordinates of the hand components at the second position. One or more components of this transformation vector may be used to modify the synthesis of sound, as described herein.

[0037] Similar to the step of receiving first information as described herein, the step of receiving second information may also include controlling one or more cameras to record one or more images of a first user's hand and processing one or more images to extract second 3D coordinates corresponding to the transformed position of the hand. The transformed 3D coordinates may be the same as or similar in form to the 3D coordinates of the first information, and they may be combined to calculate a transformation vector relating to the transformation between the first and second positions of the user's hand.

[0038] Figure 9 illustrates a keyboard being played by a user, with the 3D coordinates of one or more hand features superimposed on the illustrative diagram. A depiction of a bounding box region 910, which may be used in several methods of a further embodiment in which the first information and note are correlated, is also superimposed on the illustrative diagram. As illustrated in Figure 10, the step of correlating the first information and note output may comprise defining a bounding box region 910 corresponding to a key identifier in 3D coordinates, comparing the bounding box region 910 to a first position of one or more hand components, and determining which of the one or more hand components is most likely to be associated with the bounding box region 910. The representation of the bounding box region 910 is shown as a dashed line in Figure 9. The bounding box region 910 may be defined as a volume in 3D space or as a discrete surface corresponding to a key 106 on a keyboard 104. The bounding box region 910 may be defined to cover the entire playing surface of a single key, or the volume in which a finger may be positioned when interacting with a key 106. The bounding box region 910 may be defined by one or more 3D coordinates corresponding to the vertices of the key 106, or by any suitable geometric structure that defines volume or playing surface. For example, a key is pressed, as shown in Figure 9. The bounding box region of this key is known and illustrated in Figure 9. The 3D coordinates of each fingertip of the user's hands are known by a method step of extracting these 3D coordinates from one or more images of the hands. In a further step of the method, the 3D coordinate of each fingertip may be compared to the 3D coordinates of the bounding box region to determine whether one of the fingertips is located within the bounding box region. Since the 3D coordinates may not be precisely aligned with the bounding box region, the method may include determining, for each component of the user's hand, the likelihood that it was a component of the hand associated with pressing a key by its proximity to the bounding box. If the most likely candidate component is identified, the identification information of this component may be provided in combination with a note-on command for further processing, analysis, or control of the instrument.

[0039] It is understood that the 3D coordinates of each component of the hand and the bounding box region may have a reference point in space corresponding to a zero position and zero angular rotation. Such a reference position may be defined in relation to a keyboard key, a playing surface of an instrument, or redefined in relation to a first piece of information. For example, when a first position is defined, that position may be the zero reference point. In relation to that reference point, a second position may be determined, and the position of the bounding box region may be determined. Alternatively, the zero reference point may be defined within the bounding box region in response to a note-on MIDI event. Or, this method may simply use a camera-referenced zero reference point maintained at a position related to one or more cameras.

[0040] Figure 11 illustrates a portion of a musical instrument synthesizer system used to calibrate the systems and methods described herein. In a first calibration step, the calibration method comprises detecting a first note from the output of a control interface corresponding to a first predefined note. This note is predefined in that the note is known as part of the method. The note may be known because it is predefined and the user is prompted to press a predefined note. In the example shown in Figure 11, the user is prompted to press a key highlighted on the screen. Alternatively, the user may be given a note identifier to press (e.g., C3 or middle C). Additionally or alternatively, the control interface may be configured to highlight a predefined note by illuminating an LED corresponding to the note. In the illustrated example, the LED is housed in a partially transparent or translucent key so that the illumination of the LED illuminates the corresponding key on the keyboard.

[0041] This calibration method further comprises receiving first information indicating a first position of one or more components of a first user's hand. The reception of the first information may be carried out according to other methods described herein.

[0042] In a further step of the calibration method, the method comprises detecting a second note in the output from a control interface corresponding to a different second predefined note. The first and second notes may be relatively close to each other on the control surface (e.g., keys in the same octave) or may be located at the far ends of the control surface (e.g., at both ends of the control surface). The method further comprises receiving second information indicating the second position of one or more components of a first user's hand. The reception of the second information may be carried out in accordance with other methods described herein in relation to the reception of the first and second information.

[0043] The method further comprises determining the position of the control interface relative to the camera based on the first and second pieces of information. Since two predefined notes are known, the position of the user's hand and fingers when these notes are played can be used to imply the position of the control surface. The method may be extendable to more than two predefined notes, and the arrangement of notes may be arranged according to the 3D coordinates of the hand components when the notes are played or the keys on the keyboard are pressed. The position of the control surface can be used to provide a zero reference point, as used in other methods described herein. The position of the control surface can provide information to enable the determination of a bounding box region for each note or key.

[0044] Optionally, this method may include determining the size of the control interface. In some examples, a keyboard may have a different number of keys, and those keys may be of varying widths (for example, a full-sized piano may have wider and more numerous keys than a portable keyboard or synthesizer). By determining the size, or by determining the width of the control interface, the bounding box area can be appropriately distributed across the field of view of one or more cameras without requiring keys at the edges of the control surface to be pressed.

[0045] Methods and systems described herein with reference to the MIDI protocol. It is understood that the scope of this disclosure encompasses any other musical communication protocol, including but not limited to OSC, MIDI polyphonic expression, or custom musical communication protocols. Sound synthesis by a synthesizer or in any other system described herein may include the generation of new musical sounds, the playback of recorded sounds, and the amplification of live-recorded sounds. The systems and methods described herein may be used in conjunction with any conventional note synthesis.

[0046] The methods and systems described herein describe implementations involving one or more cameras, but it is understood that these systems and methods may be applied using any number of cameras or any number of sensors to perform optical or physical measurements of a user within a field of view or sensor area. For example, optical measuring sensors such as laser-based optical sensors, tracking motion sensors relying on structured light or time-of-flight calculations, 3D scanners, or physical sensors attached to a hand or glove may be used to determine the movement and / or position of a user's hand.

[0047] The methods and systems described herein are described with reference to vectors encoded by referring to three position coordinates and three rotation coordinates. Without departing from the methods and systems described herein, it is understood that any suitable representation of the user's hand may be used to encode the position or movement of the hand. For example, the position and / or movement of the user's hand may be encoded in quaternion notation to represent a rotation about the origin in three-dimensional Euclidean space. The representation of the position and / or movement of the user's hand may further be constrained based on inverse kinematics; that is, the position and / or movement of one element may be constrained to another element. For example, the position of the fingertips may be encoded simply as angles relative to the finger joints, rather than using the completely unique coordinates of the fingertips.

Claims

1. A computer implementation method for controlling the output of a musical instrument, Receiving first information indicating the first position of one or more components of the first user's hand, To detect the note output from the aforementioned instrument, To identify a first component associated with the note output from one or more components of the first user's hand, the first information and the note output are correlated; To provide a first output comprising the aforementioned musical note output and the correlated first information, A computer implementation method comprising the above.

2. Synthesizing a first sound corresponding to the aforementioned note output. The computer implementation method according to claim 1, further comprising:

3. Receiving second information indicating the second position of one or more components of the first user's hand, To provide a transformation vector, the transformation between the first position and the second position is calculated, To provide a second output comprising the aforementioned transformation vector, The computer implementation method according to claim 1 or claim 2, further comprising:

4. The first sound is modified in proportion to the transformation vector. The computer implementation method according to claim 3, as dependent on claim 2, further comprising the above.

5. The step of receiving the first information is: Controlling the camera to record one or more images of the first user's hand, Processing one or more images to extract 3D coordinates indicating a first position of one or more components of the hand, wherein the first information includes the 3D coordinates, A computer implementation method according to any prior claim, comprising:

6. The computer implementation method according to claim 5, wherein the 3D coordinates indicate a first position of one or more components of the hand relative to the position of the instrument interface.

7. The computer implementation method according to any prior claim, wherein the instrument comprises a plurality of keys.

8. The computer implementation method according to claim 7, wherein detecting a note output from the instrument comprises receiving music communication protocol data from the instrument in the form of a note-on event having a key identifier, and optionally the music communication protocol data is in the form of MIDI data or OSC data.

9. Correlating the first information and the musical score output means Defining a bounding box region corresponding to the key identifier within 3D coordinates, Comparing the first position of the bounding box region and one or more components of the hand, To determine the component of the hand that is most likely to be associated with the bounding box region among one or more components of the hand, The computer implementation method according to claim 8, comprising:

10. The computer implementation method according to claim 9, comprising comparing the bounding box region and the first position of one or more components of the hand to determine the distance between the 3D coordinates of the bounding box region and the 3D coordinates of the first position of one or more components of the hand.

11. The computer implementation method according to any one of claims 1 to 6, wherein detecting the note output from the instrument comprises receiving an audio signal from a microphone or pickup configured to detect the sound emitted from the instrument.

12. The step of receiving the second information described above is: Controlling the camera to record one or more images of the first user's hand, Processing one or more images to extract transformed 3D coordinates indicating a second position of one or more components of the hand, wherein the second information comprises the transformed 3D coordinates. A computer implementation method according to claim 3, or any of claims 4 to 11 when dependent on claim 3, comprising the above.

13. The computer implementation method according to claim 12, wherein the converted 3D coordinates indicate a second position of one or more components of the hand relative to the position of the instrument interface.

14. A computer implementation method according to claim 3, or any of claims 4 to 13 as dependent on claim 3, wherein calculating the transformation between the first position and the second position comprises subtracting the first information from the second information in order to provide a transformation vector indicating the degree of movement or rotation between the first position and the second position.

15. A computer implementation method according to claim 4 or any of claims 5 to 14 as dependent on claim 4, comprising providing one or more MIDI data of polyphonic key pressure, control change, channel pressure, and pitch wheel change, for modifying the first sound in proportion to the transformation vector.

16. The computer implementation method according to claim 15, wherein the polyphonic key pressure comprises a key identifier and a pressure value corresponding to the conversion vector.

17. The computer implementation method according to claim 15, wherein the control change comprises a controller number and a new value, the new value corresponding to the transformation vector.

18. The computer implementation method according to claim 15, wherein the channel pressure comprises a pressure value corresponding to the transformation vector.

19. The computer implementation method according to claim 15, wherein the pitch wheel change includes a new value relative to the center value, and the new value relative to the center value corresponds to the transformation vector.

20. The computer implementation method according to any prior claim, wherein one or more components of the first user's hand include one or more of the first user's fingertips, finger joints, and wrist joints.

21. The computer implementation method according to any prior claim, wherein the first position comprises translation and rotation of one or more components of the first user's hand.

22. It is a synthesizer system for musical instruments, Control interface and A camera directed towards the aforementioned control interface, One or more processors that communicate with the control interface and the camera, one or more processors configured to perform the method according to any one of claims 1 to 21, A musical instrument synthesizer system equipped with [features / equipment].

23. The synthesizer system according to claim 22, wherein the control interface is a keyboard having a plurality of keys, and each of the plurality of keys optionally corresponds to a musical note.

24. The synthesizer system according to claim 22 or 23, wherein the camera comprises two sensors, each of which is optionally directed towards a different portion of the control interface, and further optionally, the different portions of the control interface overlap.

25. A method for calibrating a musical instrument synthesizer system comprising a control interface and a camera directed toward the control interface, To detect a first note in the output from the control interface that corresponds to a first predefined note, Receiving first information indicating the first position of one or more components of the first user's hand, Detecting a second note in the output from the control interface that corresponds to a different second predefined note, Receiving second information indicating the second position of one or more components of the first user's hand, Based on the first and second pieces of information, the position of the control interface relative to the camera is determined, A method that includes [a certain feature].

26. Determining the size of the control interface, wherein optionally, the size of the control interface is the width of the control interface. The method according to claim 25, further comprising:

27. The method according to claim 25 or claim 26, wherein the control interface is a keyboard, the first note corresponds to a first key on the keyboard, and the second note corresponds to a second key on the keyboard.

28. The user is prompted to press the first predefined note, The user is prompted to press the second predefined note, Furthermore, The method according to any one of claims 25 to 27, wherein prompting the user to press the first and second predefined notes comprises displaying indications of the first and second predefined notes on a screen communicating with the synthesizer, or prompting the user to press the first and second predefined notes comprises illuminating corresponding indicator lights located near the first and second keys.

29. The method involves defining a bounding box for each key of the aforementioned keyboard, wherein the bounding box for each key corresponds to the outer edge of each key. The method according to any one of claims 25 to 28, further comprising:

Images