Generating corrected hand position using hand detection system

Abstract

A method for generating a corrected position of a hand using a hand detection system. The method comprises using a sensor, which may be a camera, an infrared sensor or lidar, to detect a first positio

Description

Technical Field The present disclosure relates to hand detection and position correction systems. Specifically, it pertains to methods and systems for determining and correcting the position of a hand in relation to a user interface device using sensor-based detection. These techniques are applicable in various fields, including but not limited to musical instruments, human-computer interaction, virtual and augmented reality systems, and ergonomic interface design. Background Hand position detection technologies have seen a range of implementations across various domains such as musical instruments, gaming, and virtual reality systems. These systems typically employ sensors in the form of cameras to capture and analyze hand movements and positions. The general principles involve processing the data captured by these devices to interpret the spatial orientation and motions of the hand. Various algorithms and methods are used to translate this data into meaningful inputs for different applications. One challenge in this field is the accurate detection of precise hand positions with respect to a user interface device being operated by the user. Misalignment between the detected and actual hand positions can occur, leading to errors in the system's response. For this reason, conventional hand detection systems require the sensor to be positioned and oriented in a specific arrangement with respect to the user interface device so that the hands and user interface device appear as expected relative to one another. Environmental factors such as lighting conditions and the presence of obstacles can further affect the accuracy of detection. These inaccuracies can cause disruptions in applications that rely on precise hand positions, such as virtual musical instruments or interactive gaming systems. Real-time correction of hand positions is an aspect for user applications that require immediate feedback. However, current technologies face limitations in achieving the desired level of real-time accuracy. This is especially relevant in scenarios where the user's actions need to be mirrored instantaneously by the system, such as in real-time musical performance or responsive virtual environments. The inability to provide accurate real-time corrections can reduce the effectiveness of these applications. Variability in hand detection accuracy across different users presents another challenge. Factors such as hand size, movement speed, and individual hand characteristics can lead to inconsistent detection results. Additionally, maintaining consistent accuracy across various applications poses difficulties due to differing system requirements and environmental conditions. Machine learning techniques have been explored to address these challenges, offering potential improvements in detection consistency by adapting to the unique characteristics of individual users and varying conditions. Summary According to a first aspect, there is provided a method of generating a corrected position of a hand. The method involves configuring a hand detection system to determine, via a sensor, a first position indicative of the position of a hand. The system determines that the hand is in contact with a user interface device and determines a second position indicative of the contact position. If the system identifies misalignment between the first and second positions, it generates a corrected first position such that both positions are aligned. Determining that the hand is in contact with the user interface device may involve the hand detection system receiving a signal indicative of user interaction. This signal can be an electrical signal corresponding to a musical note, which may include a note-on event or a MIDI (Musical Instrument Digital Interface) interface signal, or it could be a sound waveform. The user interface device may be a keyboard or a stringed musical instrument such as a guitar, harp, or chordophone. The sensor may be a camera, an infrared sensor or lidar sensor. The sensor may be part of the hand detection system or separate and coupled to the hand detection system. In some examples, the sensor may be in a mobile computing device. Determining the first position indicative of the hand's position may involve identifying an object in sensor data that may be a hand. A classification procedure may be performed to classify the object as a hand. This can include using one or more machine learning models on the sensor data. Determining that the hand is in contact with the user interface device may include determining that at least one finger of the hand is in contact. This determination can involve assigning a probability score to each finger based on sensor data, with a score above a threshold indicating contact. Alternatively, or additionally, it may involve noting that the hand has been in contact for a period and that at least one finger has moved less than a threshold distance during this time. The second position can be detected directly via the sensor or retrieved from storage, corresponding to a previously detected position. The sensor may include a camera, and misalignment determination might entail correcting the first position's coordinates to account for lens distortion, generating a third position for comparison. Coordinates of the third position can be generated by applying a function to the first position's coordinates, which may involve multiplying a coordinate by a scalar. The function may be determined experimentally and / or via one or more suitable algorithms. The process of determining misalignment between the first and second positions may include comparing their coordinates and establishing that the difference exceeds a threshold. The coordinates of the first position may correspond to a finger of the hand, while the coordinates of the second position may correspond to an input element of the user interface device, such as a musical key on a keyboard or a string on a stringed instrument. The coordinates of the first and second positions may include respective x and y coordinates, with the x-axis aligning with the largest dimension of a keyboard or guitar neck. The coordinates of the first and second positions may also include a z coordinate. The coordinates may therefore provide a three-dimensional representation of the first and second positions, allowing for correction of the first position in three-dimensions. The threshold can be based on the detected size of the hand, such as the length of a finger. The threshold may be learned by a machine learning model or determined experimentally. Generating a corrected first position may involve adjusting the orientation of the first position to ensure the difference between its coordinates and those of the second position is below a threshold. In configurations where the hand detection system includes a foldable device, the sensor may be positioned at a fixed distance from a pivot point, and the orientation adjustment may involve rotating the first position about this pivot point. Alternatively, generating a corrected first position may involve adjusting, for example translating, the first position to meet the alignment criteria. The method may include displaying the corrected first position on a display, optionally showing an animated transition from the first to the corrected first position. The second position may also be displayed. Additionally, at least one of the positions can be stored in association with a sensor identifier to a remote device or local storage. The method may include transmitting one or more of the first, second, and corrected first positions to a remote device, which could be a mobile phone, tablet, personal computer, or server, connected via a local connection or a wide area network. According to a second aspect, there is provided a hand detection system configured to perform the method according to the first aspect. The hand detection system may comprise one or more processors and a memory, with the memory including instructions to execute the method. In some examples, the hand detection system may be a distributed or centralized system. The hand detection system may comprise a local or remote computing device comprising, or at least coupled to, a sensor near the user interface device. Additionally, or alternatively, the hand detection device may include the user interface device. According to a third aspect, there is provided one or more computer-readable media including instructions that, when executed by one or more processors, cause them to perform the method according to the first aspect. Brief Description of the Figures Examples of the present disclosure will now be described with reference to the accompanying drawings: Figure lisa flow diagram showing a method for generating a corrected position of a hand; Figure 2 is a flow diagram showing steps that may be performed in part 108; Figure 3 is a flow diagram showing steps for determining that the hand is in contact with the user interface device; Figure 4 is a flow diagram showing an example of how part 104 can be performed; Figure 5 is a flow diagram showing an example of how part 108 is performed when the sensor is a camera with a lens; Figure 6 is a plan view showing the detection of a hand position and the determination of a contact position on an electronic keyboard according to an example; Figure 7 is a plan view showing the scenario after correcting the first position on an electronic keyboard; and Figure 8 is a schematic diagram showing an arrangement of a hand detection system, user interface device and a sensor. Detailed Description Generally, it is difficult to align a position of a hand detected via a sensor with a position at which it is determined that the hand is in contact with a user interface device. This may be because the relative position between the hand and user interface device will appear to change as the relative position and angle of the sensor changes. For instance, at a first time, the sensor may be associated with a first position and orientation with respect to the user interface device. At the first time, a hand detection system may determine the positions of the hand and a position at which the hand contacts the user interface device using data generated by the sensor. In this case, the positions of the hand and contact with user interface device may be approximately aligned since they are determined from the same sensor data (i.e. from a single field of view). At a second time, the sensor may have been moved, and so will now be associated with a second position and orientation with respect to the user interface device. At least one of the second position and orientation may differ from the first position and orientation. At the second time, the hand detection system may again determine the position of the hand and the position at which the hand contacts the user interface device. The hand detection system may use the position at which the hand contacts the user interface device from the first time, since it may be more unlikely that the user interface device has moved with respect to the sensor than the hand, and so continuous identification and classification of the user interface device may be computationally inefficient. However, in situations when the sensor has moved between the first and second time, the determined hand and contact positions will typically not be aligned. The techniques described herein address this problem by generating a corrected position of the hand that aligns with the position at which the hand contacts the user interface device. Method for Generating Corrected Hand Position Figure lisa flow diagram showing a method for generating a corrected position of a hand, referred to as method 100. The method involves several steps executed by a hand detection system. The first step, referenced as 102, involves determining a first position indicative of a hand's position via a sensor. The sensor can be part of the hand detection system or a separate component coupled to it. Various types of sensors can be used, such as optical, infrared, or ultrasonic sensors. The first position might be received directly from the sensor or generated from the data collected by the sensor. It can be an average position of the hand, such as a centroid of the hand's shape, an average position of the fingers, or a specific point like a fingertip likely in contact with a user interface device. The first position may be defined as a point, an area, or a 3D volume corresponding to the space occupied by the hand. Identification and classification of the hand based on sensor data may be performed using any suitable algorithm or technique such as template matching, histogram of oriented gradients (HOG), convolutional neural networks (CNNs), regionbased convolutional neural networks (R-CNN), You Only Look Once and Mask R-CNNs. Feature extraction methods may identify key points on the hand, such as joints or fingertips. Skeletal tracking algorithms can be used to estimate the hand's pose and position in 3D space. Next, the method includes determining that the hand is in contact with a user interface device, as referenced by 104. This can involve the hand detection system receiving a signal indicative of the user's interaction with the interface. Examples of user interface devices include musical instruments like pianos, guitars, harps, and chordophones. Other examples include gaming systems with one or more controllers. The user interface device may comprise an electronic device that can be coupled to the hand detection system. The coupling may be wired or it may be wireless. Reliable contact detection may be achieved by receiving a signal indicative of user interaction with the user interface device. This ensures that the system accurately identifies when the hand is in contact with the device. Consequently, the system can reliably differentiate between intentional and unintentional interactions. The signal may be an electrical signal corresponding to a musical note. This enables the hand detection system to seamlessly integrate with musical applications. This allows for real-time interaction and feedback, enhancing the user experience. Musicians can utilize the system to create, modify, and control musical compositions efficiently. The signal may comprise a MIDI interface signal which may facilitate standardized signal processing. This ensures that the system may seamlessly interact with a variety of musical instruments and devices. As a result, users can integrate the system into existing setups without compatibility issues. The signal may comprise a sound waveform. The sound waveform can be recorded, for example, by a microphone or other suitable sound recording device, which converts acoustic waves into electrical signals. These signals can then be digitized and processed to detect user interaction with the interface device. The capability to process sound waveforms may allow the system to manage a variety of audio signals. This enhances the system's applicability across different audio-related tasks. It can be used in diverse environments, from musical applications to voice recognition systems. Following this, the method involves determining a second position indicative of where the hand contacts the user interface device, as shown in block 106. The second position may be detected using the sensor or retrieved from storage. If detected via the sensor, it can be computationally intensive as the system must identify and classify both the hand and the user interface device. Alternatively, retrieving the position from storage is more efficient but may be more likely to lead to misalignment if the sensor's position or orientation changes. The second position can be a point or an area and may represent an average position of the contact. The method then checks if the first and second positions are misaligned, as represented by decision block 108. This step may involve determining if the difference between the coordinates of the first and second positions exceeds a threshold. An example of misaligned positions is illustrated in Figure 6, and this determination process is detailed further in method 200 shown in Figure 2. If the positions are misaligned, the method proceeds to generate a corrected first position, as shown in block 110. This could involve adjusting the orientation of the first position or translating it so that the difference between its coordinates and the second position's coordinates is below a threshold. The corrected first position might be displayed on a display, such as display 814 shown in Figure 8. The display may show an animated transition from the first position to the corrected first position, and both positions may be displayed together for the user to view. At least one of the first, second, and corrected positions can be stored with an identifier of the sensor. Additionally, or alternatively, at least one of the first, second, and corrected positions may be transmitted to one or more remote devices, such as mobile phones, tablets, personal computers, or servers via local or wide area networks. If the first and second positions are not misaligned, the method concludes that no correction is required, as indicated by block 116. The process aims to ensure that the detected hand position aligns with the actual position of contact on the user interface device, thereby improving accuracy in various applications. The described method enhances the accuracy of hand position detection by correcting the position determined via the sensor to align with the actual contact position on the user interface device. This correction ensures that any discrepancies between the detected position and the actual contact point are minimized. As a result, the system provides a more precise representation of the hand's location. In other words, this reduces errors in hand position detection, particularly where the sensor is moved and / or reoriented during use, or in between uses, of the user interface device. Aligning the corrected hand position with the actual contact point improves the user's interaction with the user interface device. This alignment ensures that the user's actions are accurately reflected, enhancing the overall experience. For instance, in the context of a musical instrument, this method allows users to analyze their performance with greater precision. This can be used to improve a user's playing technique by allowing the user to review the positioning of the hand while playing a musical piece. For instance, the method can help a user improve transitioning between notes and / or improve fingering technique. For example, the ability of the hand detection system to accurately determine a corrected hand position allows for an improved analysis of the performance. The method supports real-time correction of the hand position, providing immediate feedback to the user. This real-time adjustment ensures that any misalignment is promptly addressed, allowing for continuous and accurate interaction. Consequently, users receive instant and reliable information about their hand positioning. Continually detecting the position of the user interface device via a sensor as well as the position of the hand can be computationally intensive. The method allows the use of a previously determined position of the user interface device to be used. In this way, the method is a calibration of the hand detection system for a particular user interface device and sensor arrangement. Process for Evaluating Positional Alignment Figure 2 relates to the features shown in Figure 1 and provides a flow diagram illustrating the steps that may be performed as part of block 108 in the method 100. The method 200 provides an example of how it can be determined whether the first and second positions are misaligned. In method 200, the initial step involves determining the coordinates of the first position, as shown in block 202. This relates to block 102 from Figure 1, where the coordinates of the first position may correspond to those of a finger of the hand. The first position may be represented by one, two, or three coordinates, representing the position in ID, 2D, and 3D space, respectively. In some examples, the coordinates of the first position comprise x and y coordinates. The x and y axes can be defined in various ways, but in some examples, it may be useful to align one of the axes, such as the x-axis, with the largest dimension of the user interface device. For instance, where the user interface comprises a piano, the x-axis could be aligned with the length of the keyboard, while for a guitar, the x-axis might align with the length of the guitar neck. The origin of the coordinate system may be at a centre of the user interface device relative to the field of view of the sensor. In some examples, the origin may be at a centre of the field of view of the sensor itself. Aligning the x-axis with the largest dimension of the user interface device provides a clear and consistent reference for positional coordinates. This alignment simplifies the mapping of hand positions relative to the device. Users can easily understand and adjust their hand positions based on this consistent reference framework. Next, block 204 corresponds to block 104 in Figure 1 and involves determining the coordinates of the second position. The coordinates of the second position may correspond to those of an input element of the user interface device. The user interface device could be a keyboard, where the input element is a musical key, or a stringed musical instrument, where the input element is a string. In some other examples, the user interface device may comprise a controller of a gaming system where the input element is a button on the controller. The coordinates of the second position are defined in the same coordinate system as the coordinates of the first position. Decision block 206 involves comparing the coordinates of the first position with those of the second position. This comparison may comprise calculating a difference between the coordinates of the first and second position, for example, by calculating the Euclidean distance between the coordinates. Where the comparison comprises calculating a Euclidean distance between the first and second positions, the threshold may be a threshold distance. This comparison helps to determine if the positions are misaligned by assessing whether a difference between the coordinates exceeds a specific threshold. Decision block 208 further evaluates whether this difference exceeds the threshold. The threshold could be determined experimentally or through machine learning techniques to ensure accurate results. Using a machine learning model to learn the threshold enables the system to adapt over time. This adaptability enhances the system's accuracy in recognizing and responding to various user inputs. Consequently, the hand detection system may become more responsive to different users and changing conditions, improving overall performance. In some examples, the threshold may be based on the detected size of the hand, such as the length of a finger. Determining the threshold based on the detected size of the hand may allow for adaptation to different users. This adaptability enhances usability by accommodating various hand sizes, ensuring a more personalized and comfortable interaction. The system can adjust dynamically, providing a more intuitive and user-friendly experience. If the difference between the coordinates does not exceed the threshold, as indicated by the 'No' strand 216, the positions are considered to be aligned. This scenario is represented by block 212, which corresponds to the 'No' strand 114 in Figure 1. Conversely, if the difference exceeds the threshold, as indicated by the 'Yes' strand 214, the positions are deemed misaligned. This situation is depicted in block 210, which corresponds to the 'Yes' strand 112 in Figure 1. In this situation, a corrected first position is generated. The method described ensures precise detection of misalignment by comparing the coordinates of the first and second positions. This comparison allows for an accurate assessment of positional discrepancies, ensuring that any deviation beyond a predefined threshold is reliably identified. Such precision is beneficial in applications requiring exact alignment, enhancing overall performance and user experience. Procedure for Determining Hand Contact with User Interface Figure 3 is a flow diagram showing steps for determining that the hand is in contact with the user interface device. This figure relates to the features shown in Figure 1 and provides a detailed view of how block 104 in method 100 from Figure 1 can be performed. Method 300 demonstrates an example of how it can be determined that the hand is in contact with the user interface device. The method 300 includes receiving data from the sensor, as indicated by block 302. This sensor data is generated by various types of sensors, such as sensors that detect electromagnetic radiation, such as optical sensors, infrared sensors and lidar sensors, or capacitive touch sensors. For instance, if the sensor is a camera, the data may be image data representing the hand's position relative to the user interface device. Other examples of sensor data include electrical signals from capacitive touch sensors or distance measurements from ultrasonic sensors. The method 300 involves receiving a signal from the user interface device, as indicated by block 304. This signal may be indicative of user interaction with the user interface device. In some examples, the signal is an electrical signal corresponding to a musical note. This may include a note-on event, which indicates when a note starts playing. The signal may be a MIDI signal, which is a standardized protocol for communication between electronic musical instruments and devices. Receiving the signal from the user interface device may occur before the data from the sensor is received at block 302, and in some examples may cause block 302 to be performed. In that sense, block 304 may initiate the rest of the method 300. Next, in block 306, a probability score is assigned to each finger of the hand based on the received sensor data and the signal received from the user interface device. This may involve the use of a machine learning model to analyze the sensor data and determine the likelihood that each finger is in contact with the user interface device. The machine learning model may be trained to recognize hand poses and / or finger positions from the sensor data. The machine learning model may analyze features such as finger orientation, proximity to the user interface device, and historical interaction patterns to assign these probability scores, thus improving the accuracy of the probability scores assigned. Analysis of the sensor data may be used alongside the received signal to assign the probability score Following this, decision 308 involves determining whether the probability score for each finger is above a predefined threshold. The threshold may be determined through various means to improve accuracy. For example, it may be set experimentally by analyzing user interactions and adjusting the threshold to minimize false positives and false negatives. Alternatively, machine learning techniques may be employed to dynamically adjust the threshold based on factors such as individual user behavior, environmental conditions, or the specific type of user interface device. If the probability score for a finger is above the threshold, the method proceeds along the yes branch 310 to block 312, where it is determined that the finger is in contact with the user interface device. This information can be used to refine the hand position detection and improve the accuracy of interactions with the user interface device. Conversely, if the probability score is below the threshold, the method moves along the no branch 314 to block 316, indicating that the finger is not in contact with the user interface device. These determinations help in accurately identifying which fingers are interacting with the user interface device, thus allowing for a more precise and responsive system. By using this probabilistic approach, the method 300 can account for variations in sensor data quality and user interaction styles, providing a robust means of determining hand / finger contact with the user interface device. This contributes to the overall accuracy and reliability of the hand position correction process described in the method 100 of Figure 1. The use of probability scores for assessing finger contact enables a more refined and accurate determination of user interaction. By assigning a likelihood value to each finger's contact state, the system can account for varying degrees of certainty in sensor data interpretation. This approach allows for a more nuanced understanding of user input, potentially reducing errors and improving the overall responsiveness of the user interface device. Example Methodfor Identifying Contacting Finger Figure 4 relates to the features shown in Figure 1, specifically illustrating an example of how determining that the hand is in contact with a user interface device can be performed. Figure 4 is a flow diagram depicting a method 400 for determining which finger or fingers of a hand are in contact with a user interface device. This method 400 may be utilized as part of step 104 in the method 100 shown in Figure 1. The process begins with decision block 402, in which it is determined whether the hand is in contact with the user interface device. The determination may be made by receiving a signal from the user interface device indicative of user interaction. The signal may indicate that a button on the user interface device has been pressed, for example. If the hand is not in contact, the method proceeds to block 410, where it is determined that the hand is not in contact with the user interface device. If the hand is in contact, the method moves to block 404, which involves determining which one or more fingers are most likely to be in contact with the user interface device. In some examples, the determination may involve analyzing fingertip data from the sensor. For instance, for each frame it may be determined which fingertip is closest to the user interface device, or closest to a pressed button on the user interface device. Where the user interface device is a piano, this may involve determining which fingertip is lowest while playing the piano. In some examples, the method 300 shown in Figure 300 may be used to perform at least some aspect of block 404. If it is determined that there are multiple points of contact between the user and the user interface device, such as where multiple buttons are being pressed, the closest fingertips and their relative positions to each other may be determined in order to match those fingertips to the pressed buttons. Following this determination, the method moves to block 406, where the movement of one or more fingers is determined via the sensor. Fingertips can be tracked using sensor data from the sensor received over a period of time. For instance, computer vision techniques may be employed to identify and track the positions of fingertips across consecutive frames of image data. Block 408 determines if one or more fingers are in contact with the user interface device. This determination may involve analyzing the movement determined in block 406. Probabilities may be incremented over a period of time, based on the determined movement of the one or more fingers. For example, if a first fingertip remains approximately stationary while a button of the user interface device is determined to be pressed, and the other fingertips move a distance greater than a threshold distance during this time, it can be established with greater confidence that the first fingertip is the finger in contact with the button. This method 400 provides a detailed approach for determining whether a hand is in contact with a user interface device and, if so, which specific finger or fingers are making that contact. By analyzing both the position and movement of fingertips over time, the system can make increasingly accurate determinations about user interactions with the user interface device. The method 400 incorporates both temporal and spatial factors. By assessing these factors, the system can more reliably identify stable contact between one or more fingers and the user interface device. This approach helps distinguish between intentional, sustained interactions and brief or accidental touches. The combination of time-based and movement-based criteria enhances the accuracy of contact detection, potentially reducing false positives and improving the overall responsiveness of the user interface device. Method for Evaluating Positional Accuracy Figure 5 relates to the features shown in Figure 1 and is a flow diagram illustrating method 500. Method 500 exemplifies the process of determining whether the first and second positions are aligned, similar to step 108 in method 100 depicted in Figure 1. In this example, the sensor may be a camera equipped with a lens, where the lens can introduce distortion into the sensor data, leading to a less accurate determination of the first position. This aspect may be significant when the user interface device is a keyboard, as lens distortion could impact accuracy, particularly towards the edges of the keyboard where the effect of lens distortion is more pronounced. Method 500 begins with block, referenced as 502, where the coordinates of the first position are corrected to address lens distortion and generate the coordinates of a third position. The coordinates of the third position may be generated by applying a function to the coordinates of the first position. This function is determined or selected specifically to mitigate the effects of lens distortion. The function may be linear, for example, involving the multiplication of at least one coordinate of the first position by a constant scalar. This may be useful where the origin of the coordinate system is at a centre of the user interface device from the point of view of the sensor, and the centre of field of view of the sensor approximately aligns with the centre of the user interface device. The function may be determined experimentally or via machine learning techniques. Following the generation of the third position coordinates, block, referenced as 504, involves comparing the coordinates of the third position with those of the second position. This comparison is critical to ascertain the alignment between the two positions. Subsequently, decision, labeled as 506, evaluates whether the difference between the coordinates of the third position and those of the second position exceeds a predefined threshold. This decision point determines the subsequent step in the process. If the difference is found to exceed the threshold, condition, referenced as 512, is marked as 'Yes, ' leading to block 508. In block 508, it is determined that the first and second positions are misaligned. On the other hand, if the difference does not exceed the threshold, condition, referenced as 514, is marked as 'No,' leading to block 510, where it is concluded that the first and second positions are aligned. Correcting coordinates to reduce lens distortion ensures more accurate position detection. This correction improves the reliability of the position determination process, particularly in devices where precision is necessary. By mitigating the distortion, the system can more effectively compare positions and make accurate alignment decisions. Detection and Determination of Hand and Contact Positions Figure 6 relates to the features shown in Figure 1 and provides a plan view illustrating the detection of a hand position and the determination of a contact position on an electronic keyboard. Figure 6 showcases an example of detecting a first position 608 by a sensor 606. The first position 608 is indicative of a position of a fingertip of the hand 604, which the system determines to be likely pressing a key. This determination may be carried out using a method similar to method 400 shown in Figure 4. The electronic keyboard 602, depicted in Figure 6, comprises a plurality of keys and serves as the user interface device. The sensor 606 is positioned in the vicinity of both the keyboard 602 and the hand 604. It is oriented to ensure that both the hand and the keyboard fall within its field of view. The sensor generates data that can be processed to classify at least the hand 604, allowing for accurate position detection. Figure 6 also includes the determination of a second position 610, which corresponds to the particular key of the keyboard 602 that the hand 604 is in contact with. This second position 610 is indicative of a position at which the hand makes contact with the keyboard, corresponding to a specific musical note. The figure illustrates how misalignment may occur between the detected first position of the hand and the detected second position of contact between the hand and the keyboard. This misalignment can affect the accuracy of the detected hand position relative to the actual key being pressed. In summary, Figure 6 demonstrates the interaction between the hand 604, the sensor 606, and the electronic keyboard 602. It highlights the potential misalignment between the first position 608 detected by the sensor and the second position 610, which corresponds to the actual contact point on the keyboard. This example underlines the importance of accurately determining both the fingertip position and the contact position to ensure precise interaction with the electronic keyboard. Corrected Hand Position Alignment Figure 7 relates to the features shown in Figure 6 and provides a plan view illustrating the scenario after correcting the first position on an electronic keyboard 602. Using the method 100 depicted in Figure 1, a corrected first position 708 has been generated, aligning with the second position 610. This alignment ensures that in a visual representation, the hand 604 and the user interface device 602 appear in the correct relative location. In Figure 7, the user interface device 602 can be seen in conjunction with the hand 604 and the sensor 606. The sensor 606 captures data indicative of the position of the hand 604 relative to the user interface device 602. Initially, a first position 608 is detected, which may not accurately reflect the actual contact point on the user interface device 602. By applying the method 100, a corrected position 708 is determined. This corrected position 708 aligns with the actual position where the fingertip of the hand 604 contacts the user interface device 602. In this example, the corrected first position 708 may be generated by translating the first position 608 so that the distance between the first and second positions is below a threshold. The translation will remain substantially accurate as the user moves their hand relative to the user interface device, since the correction determining a systematic difference caused by movement of the sensor. This means that the hand detection system does not need to continually determine a correction to apply to the first position, increasing efficiency and allowing for accurate hand detection even when the sensor does not have a fixed position and / or orientation with respect to a user interface device. The corrected position 708 demonstrates the effectiveness of the applied method in aligning the detected fingertip position with the actual contact point on the user interface device 602. The corrected position 708, as shown in Figure 7, confirms that the fingertip's detected position matches the point of contact. Arrangement of Hand Detection System and Sensor Figure 8 relates to the features shown in Figures 1, 2, 3, 4, 5, 6, and 7. It provides a schematic diagram illustrating an arrangement of a hand detection system according to an example. The arrangement includes a computing device 808, a sensor 806 and a user interface device 804. The arrangement may be used to perform one or more of the methods shown in Figures 1 to 5, as well as generate a corrected first position as shown in Figures 6 and 7. The hand detection system 808 is configured to determine the position of a hand 802 in contact with a user interface device 804. The hand detection system comprises at least the computing device 808. In some examples, it can include mobile devices such as smartphones or tablet computers. In these cases, the sensor 806 and display 814 may be incorporated into the same device as the computing device 808. In other words, the hand detection system may include at least one of a sensor and a display in some examples. Furthermore, one or more components of the hand detection system could be located remotely from the user interface device. For instance, at least one of the one or more processors 810 and the storage 812 may be located remotely, for example, in one or more cloud computing devices. In some configurations, the computing device 808 may include a foldable device, such as a foldable smartphone, a laptop computer, a dedicated foldable hand detection system, or a mobile device coupled to a foldable stand case. Integrating the computing device into a foldable device allows for enhanced flexibility and portability. This configuration enables the hand detection system to be easily transported and deployed in various environments. The foldable nature of the device ensures that it can be compactly stored when not in use. The sensor 806 is shown as coupled to the computing device 808. However, it is understood that in some examples, the computing device 808 may include the sensor. The sensor 806 generates data indicative of the position of the hand 802 relative to the user interface device 804, which in this example is a keyboard. It is understood to be any suitable user interface device. The display 814 may show an animated transition from the first position to the corrected first position. Examples of image displays include LCD screens, OLED panels, and e-ink displays. The display 814 may be incorporated into the same computing device as the hand detection system, or it may be provided separately from the hand detection system. The one or more processors 810 of the computing device 808 execute instructions to perform various functions. Examples of processors include central processing units (CPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs) field programmable logic arrays (FPGAs), digital signal processors (DSPs), and general purpose microprocessors. A processor typically executes instructions to process data, control peripheral devices, and perform complex calculations. The storage 812 may comprise computer-readable media that store instructions. The computer-readable media may be a tangible, non-transitory computer readable media. When executed by the one or more processors 810, these instructions cause the processors to perform certain functions. Examples of computer-readable media include solid-state drives (SSDs), hard disk drives (HDDs), RAM, ROM, CD-ROMs, and memory cards. The storage 812 can hold data such as the first position, the second position, and the corrected first position, potentially in association with an identifier of the sensor 806, and transmit this data to remote devices or local storage within the hand detection system. The animated transition from the first position to the corrected first position enhances visual feedback for users. This clear and intuitive display allows users to easily understand the correction process. It provides a seamless and engaging experience, improving user interaction with the system. Storing the first position, second position, and corrected first position in association with a sensor identifier ensures comprehensive data logging for future reference or analysis. This allows for detailed tracking of hand movements and positions over time, facilitating the identification of patterns or anomalies. The stored data can be utilized for improving the accuracy and functionality of the hand detection system through iterative updates and refinements. Variations The user interface device is illustrated in Figures 6, 7 and 8 as a keyboard. However, it is understood that the user interface device can be any suitable device, including but not limited to different musical instruments, such as a piano, guitar, harp, or chordophone, a controller for a gaming system, or some other electronic user interface device. The hand detection system may comprise one or more computing devices. It can include a mobile device such as a smartphone or a tablet computer. In these configurations, the sensor 806 and the display 814 may be incorporated into the same device as the hand detection system. In such examples, the hand detection system will typically be in the vicinity of the user interface device during operation. Alternatively, one or more components of the hand detection system may be located remotely from the user interface device. For instance, the one or more processors 810 and / or storage 812 could be situated in one or more cloud computing devices. The hand detection system may comprise a foldable device, such as a foldable smartphone, a laptop computer, a dedicated foldable hand detection system, or a mobile device coupled to a foldable stand case. The first position refers to the position of the hand as determined via the sensor. This position may be the location of a fingertip of the hand, which may be identified as the fingertip most likely pressing a key. Methods such as the one illustrated in Figure 4 can be used to determine this. The second position may correspond to a particular key of the keyboard, which in turn corresponds to a specific musical note. In Figure 8, the sensor 806 is shown as being coupled to the hand detection system 808. Nevertheless, it is understood that in some examples, the hand detection system may include the sensor. For example, in the configuration where the sensor 806 is positioned in the vicinity of the keyboard 804 and the hand 802, it is oriented such that both the hand and keyboard are within the sensor's field of view. This sensor generates data that can be processed to classify at least the hand 802. In some examples, there may be more than one sensor used to determine the first position. Where this is the case, the sensors used may be of different types, such that different types of sensor data are combined to determine the first position. The display 814 can be incorporated into the same computing device as the hand detection system or can be provided separately. Examples of image displays include monitors, touchscreen interfaces, or projection screens. The one or more processors 810 can be any suitable processing units such as central processing units (CPUs), graphics processing units (GPUs), or digital signal processors (DSPs). These processors execute instructions to perform various tasks including processing sensor data, determining the positions of the hand and user interface device, and generating corrected positions. Further Variations and Advantages Mapping the coordinates of the finger to the input element allows for a clear and direct correlation between user actions and interface responses. This ensures that the system accurately interprets the user's intended actions, enhancing the overall user experience. The precise mapping reduces the likelihood of errors, making the interaction more intuitive and efficient. Identifying the input element as a musical key or string allows for precise interaction with the specific type of user interface device. This specificity ensures that the system can accurately interpret user inputs based on the type of instrument being used. Consequently, the system can provide more accurate feedback and interaction tailored to the particular musical instrument, enhancing the overall user experience. Enhanced positional accuracy is achieved by adjusting the orientation of the first position. This adjustment ensures that the detected hand position aligns more closely with the actual key being pressed. This leads to more precise identification of the intended musical note. Threshold-based correction allows for customizable accuracy levels. By setting a threshold for positional differences, the system can be tailored to meet specific application requirements. This flexibility ensures that the positional correction can be adapted to various user needs and contexts. Positioning the sensor at a fixed distance from the pivot point maintains consistent and reliable detection. This setup ensures that the sensor's position remains stable regardless of the device's orientation. As a result, the hand detection system can accurately track hand movements and positions in different configurations. Translating the first position to reduce coordinate differences enhances the accuracy of the corrected position. This translation ensures that the corrected first position closely aligns with the second position, minimizing any discrepancies. As a result, the system can more precisely identify the intended key or note, improving the overall performance of the hand detection system. Determining contact based on at least one finger allows for more granular detection of user interactions. This approach enhances the precision of identifying which specific part of the hand is engaging with the user interface device. Consequently, it enables more accurate interpretation of user inputs, improving the overall responsiveness and functionality of the system. Detecting the position of the user interface device via the sensor enables real-time tracking of the device's location. This allows for dynamic adjustments based on the current position, enhancing the user experience. The system can respond promptly to changes, ensuring accurate and timely feedback. Utilizing historical position data allows the system to recognize and respond to familiar patterns. By retrieving previously detected positions from a storage device, the system can enhance its accuracy in identifying recurring hand positions. This capability can improve the overall user experience by providing more consistent and reliable responses. Incorporating a camera as the sensor allows for detailed visual detection and analysis of the user interface device's position and user interactions. The camera can capture high-resolution images, enabling precise tracking of hand movements and finger placements. This visual data can be processed to enhance the accuracy of the system in identifying and responding to user inputs. Displaying the second position on a display enables users to immediately view the detected position. This feature enhances the interaction by providing real-time feedback. Consequently, it improves overall usability. Automated position generation allows for the coordinates of the third position to be determined by applying a function to the coordinates of the first position. This method facilitates a streamlined process, reducing the need for manual input. It ensures that the position determination is consistent and reliable. Multiplying a coordinate by a scalar determined experimentally allows for finetuning across various devices. This approach ensures that the method can be adapted to different dimensions, such as those of keyboards or guitars. This adaptability enhances the system's flexibility in accommodating diverse user interface devices. The experimental determination of the scaler provides a tailored fit for each specific device. This customization ensures that the coordinate adjustments are precise, aligning with the unique characteristics of each instrument. Consequently, this method supports a wide range of applications, from musical keyboards to stringed instruments like guitars. The method's compatibility with various musical instruments such as pianos, guitars, harps, and chordophones enhances its versatility. This broad compatibility allows the method to be applied across different types of user interface devices, making it adaptable to various musical contexts. Consequently, users can employ the method with their preferred instrument, facilitating a more personalized and flexible experience. Incorporating infrared or lidar sensors enhances the sensing capabilities of the system. These sensors provide advanced detection mechanisms that improve the accuracy of position determination. This leads to more reliable identification of the hand's position relative to the user interface device. The use of infrared or lidar sensors contributes to the system's reliability. These sensors are capable of capturing detailed spatial information, which reduces errors in position detection. This ensures consistent performance across various operating conditions. Using x and y coordinates aligned with the instrument's dimensions ensures more accurate detection and correction of hand positions. This alignment minimizes errors in identifying the location of the hand or fingers. Consequently, it enhances the precision of feedback provided to the user for improving their technique. Transmitting positions to a remote device facilitates real-time monitoring from various locations. This enables users to observe and analyze the positions without being physically present. Consequently, it enhances the flexibility and convenience of monitoring activities. The system's compatibility with various remote devices, such as mobile phones, tablets, personal computers, and servers, enhances its flexibility. Users can interact with the system using their preferred device, making it more accessible. This versatility allows for seamless integration into different environments and workflows. Flexible connectivity is achieved by allowing the remote device to connect to the hand detection system through either a local connection or a wide area network. This ensures that the communication between the hand detection system and remote devices can adapt to various environments and requirements. It provides robust and adaptable communication options, enhancing the system's versatility. Including a sensor, processors, and memory with executable instructions ensures a cohesive and efficient hand detection system capable of performing the method. This integration allows for streamlined data processing and analysis, reducing latency and improving response times. The system can quickly and accurately detect hand positions and movements, enhancing user interaction and experience. Utilizing a distributed system with local or remote computing devices allows for scalable and flexible processing capabilities. This configuration enables the system to handle varying amounts of data and computational load efficiently. It also facilitates easy upgrades and maintenance by distributing tasks across multiple devices. Coupling the sensor to a computing device in the vicinity of the user interface device ensures reliable and consistent hand detection performance. This proximity reduces the likelihood of data loss or interference during transmission. It also enhances the accuracy of hand position detection by minimizing potential delays. Having a local computing device handle sensor data can reduce latency, providing quicker response times for user interactions. This immediate processing of data allows for real-time feedback to the user. Consequently, it enhances the overall user experience by making interactions more seamless and intuitive. Incorporating the sensor into a mobile device facilitates portable and convenient hand detection. This integration enhances user flexibility by allowing the hand detection system to be used in various locations without the need for additional equipment. Users can easily carry their mobile devices, making the hand detection system accessible and practical for on-the-go use.

Claims

1. A method of generating a corrected position of a hand, the method comprising configuring a hand detection system to:determine, via a sensor, a first position indicative of a position of a hand; determine that the hand is in contact with a user interface device;determine a second position indicative of a position at which the hand is in contact with the user interface device;determine that the first position and the second position are misaligned; and generate a corrected first position such that the corrected first position and the second position are aligned.

2. The method according to claim 1, wherein determining that the hand is in contact with the user interface device comprises receiving, by the hand detection system, a signal indicative of user interaction with the user interface device.

3. The method according to claim 2 wherein the signal is an electrical signal corresponding to a musical note.

4. The method according to claim 3 wherein the electrical signal comprises a note-on event, and optionally wherein the signal is a MIDI interface signal.

5. The method according to claim 3 wherein the electrical signal comprises a sound waveform.

6. The method according to any preceding claim, wherein the user interface device is a keyboard or a stringed musical instrument.

7. The method according to any preceding claim, wherein determining that the first position and the second position are misaligned comprises comparing coordinates of the first position with coordinates of the second position, and determining that a difference between the coordinates of the first position and the coordinates of the second position exceeds a threshold.

8. The method according to claim 7, wherein the coordinates of the first position correspond to coordinates of a finger of the hand and wherein the coordinates of the second position correspond to coordinates of an input element of the user interface device.

9. The method according to claim 8, wherein the user interface device is a keyboard and the input element is a musical key, or wherein the user interface device is a stringed musical instrument and the input element is a string.

10. The method according to any of claims 7 to 9, wherein the threshold is determined based on a detected size of the hand, and optionally wherein the detected size of the hand is a length of a finger of the hand.

11. The method according to any preceding claim, wherein generating a corrected first position comprises adjusting an orientation of the first position to provide the corrected first position such that a difference between the coordinates of the corrected first position and the coordinates of the second position is below a threshold.

12. The method according to claim 11, wherein the hand detection system comprises a foldable device, and wherein the sensor is implemented in the foldable device and positioned at a fixed distance from a pivot point of the foldable device, and wherein adjusting the orientation of the first position comprises rotating the first position about the pivot point.

13. The method according to any preceding claim, wherein generating a corrected first position comprises adjusting the first position to provide the corrected first position such that a difference between coordinates of the corrected first position and coordinates of the second position is below a threshold.

14. The method according to any preceding claim, wherein determining that the hand is in contact with a user interface device comprises determining that at least one finger of the hand is in contact with the user interface device.

15. The method according to claim 14, wherein determining that at least one finger of the hand is in contact with the user interface device comprises:assigning a probability score to each of the at least one finger of the hand based on data generated by the sensor, each probability score indicating a likelihood that a respective finger is in contact with the user interface device; anddetermining that a finger having a probability score above a threshold probability is in contact with the user interface device.

16. The method according to claim 14 or claim 15, wherein determining that at least one finger of the hand is in contact with the user interface device comprises:determining that the hand is in contact with the user interface device for a period of time; anddetermining, via the sensor, that the at least one finger of the hand has moved less than a threshold distance in the period of time.

17. The method according to any preceding claim, wherein determining the second position comprises detecting a position of the user interface device via the sensor.

18. The method according to any of claims 1 to 16, wherein determining the second position comprises retrieving the second position from a storage device, wherein the retrieved second position corresponds to a previously detected second position.

19. The method according to any preceding claim, wherein the sensor comprises a camera.

20. The method according to claim 19, wherein determining that the first position and the second position are misaligned comprises:correcting coordinates of the first position to reduce an effect of lens distortion associated with the sensor, thereby generating coordinates of a third position;comparing coordinates of the third position with coordinates of the second position; anddetermining that a difference between the coordinates of the third position and the coordinates of the second position exceeds a threshold.

21. The method according to any preceding claim, further comprising displaying the corrected first position on a display by displaying an animated transition from the first position to the corrected first position.5 22. The method according to any preceding claim, further comprising displaying thesecond position on a display.

23. The method according to any preceding claim, further comprising storing one or more of the first position, the second position, and the corrected first position in 10 association with an identifier of the sensor to a remote device or a storage local to the hand detection system.

24. A hand detection system configured to perform the method according to any of the preceding claims.

25. One or more computer readable media comprising instructions that when executed 15 by one or more processors, cause the one or more processors to perform the method according to any of claims 1 to 23.

Images