Left and right manipulator identification method for multi-finger touch screen interaction

Abstract

The invention relates to a method for identifying left and right manipulators for multi-finger touch screen interaction, which comprises the following steps of: S1, acquiring touch data through an API (Application Program Interface), and identifying the number of fingers and action types based on the touch data; s2, all the fingers are rearranged, all the adjacent two fingers serve as a group, N-1 two-finger groups are obtained, and N is the number of the fingers; s3, for each two-finger group, constructing a spatial triangle based on the positions and orientations of different fingers at the same moment, extracting spatial features based on the spatial triangle, constructing a time triangle based on the positions and orientations of the same finger at different moments, and extracting time features based on the time triangle; s4, respectively inputting the time features and the space features of the two-finger groups into first classifiers with independent values to obtain a plurality of first classification results in one-to-one correspondence with the number of the two-finger groups; and S5, fusing the first classification results of all the two-finger groups to obtain a left and right hand classification result. Compared with the prior art, the method has the advantages that the accuracy can be improved under the conditions of not depending on an external sensor, not needing calibration and not needing to access a privileged API (Application Program Interface).

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and human-computer interaction, and in particular to a method for recognizing left and right hands for multi-finger touchscreen interaction. Background Technology

[0002] With the widespread adoption of touchscreen technology in mobile devices (phones, tablets) and IoT devices, touch interaction has become a mainstream method of human-computer interaction. In addition to single-finger touch, most mainstream mobile devices now incorporate multi-finger touch.

[0003] In some advanced technologies, multi-finger touch can perform up to 12 tasks. Specifically, it combines three action types (zoom in, zoom out, and swipe) and the number of fingers (two, three, four, and five fingers) in pairs to obtain a total of 12 cases, thus enabling the configuration of 12 tasks.

[0004] Those skilled in the art are not satisfied with this and hope to increase the number of tasks that can be performed by using gesture control. The conventional approach to this improvement is to increase the types of actions or to increase the classification of the number of fingers, such as distinguishing different combinations of two fingers for two fingers. However, this approach often requires a high level of user skill and is difficult to transfer to different users after adaptation.

[0005] Therefore, some engineers have proposed adding a new category—left-hand or right-hand recognition—to double the number of task types by identifying whether it's the left or right hand. However, determining the specific hand and finger behind a screen gesture is far from easy, and current mainstream solutions fall into three categories: 1. Solution based on external vision sensor: Computer vision is enabling hands to interact with touch surfaces. However, these vision-based methods require additional hardware or specific environmental conditions, such as good lighting and camera angles, which increases the complexity and cost of the system. These requirements make solutions based on external vision sensors impractical for ubiquitous touchscreen devices, as they cannot be easily integrated into everyday mobile interactions.

[0006] 2. Solutions based on additional wearable devices: Wearable sensor devices are mounted on the user to identify the operating hand or finger. For example, hand motion data when touching a screen can be collected via a wrist-worn inertial measurement unit (IMU) or biosensor wearable device to further identify the operating hand and technique. External RGB or depth camera images and their statistical characteristics can be used to track gestures and determine which finger is being used. Solutions based on additional wearable devices achieve fine-grained recognition, but they introduce significant usability friction, noticeably impairing ease of use and scalability.

[0007] 3. Touchscreen-based solutions: Recently, many researchers have focused on using touchscreens' own sensor data, without requiring any external devices, to identify active hands / fingers or their gestures. These methods can be categorized into four types: 1) Statistical feature modeling: Relying on manually designed statistical features such as trajectory length and velocity variance, it has poor robustness in complex scenarios such as multiple overlapping fingers and non-standard gestures.

[0008] 2) Time series modeling: Deep sequence learning models are used to encode the dynamic behavior of touch interaction to capture time-dimensional information such as the movement trajectory, speed change, and posture change of the finger in continuous time frames. However, the features captured by the raw time series data are often quite coarse, and a large labeled corpus is needed in practical applications to ensure robust performance.

[0009] 3) Spatial Relationship Modeling: This modeling approach is a geometry-driven method that infers hand relationships from instantaneous finger layout. For example, extended spatial modeling uses quadratic coding methods to reconstruct hand posture from multi-touch input by quantifying the geometric correlations between touch points. Spatial relationship modeling methods rely on fixed anatomical assumptions, which can lead to fragility when users adopt atypical postures.

[0010] 4) Capacitive Image Modeling: The raw sensor mesh is processed into a low-resolution image, which is then processed using a modern visual backbone network. Capacitive image modeling requires privileged hardware interfaces that are rarely exposed by the local API, and this low-level data may contain user-specific features related to user identity, potentially posing a privacy risk.

[0011] In summary, among existing technologies, there is no left-hand recognition technology that can simultaneously meet the requirements of "no additional hardware", "no calibration required", "cross-device compatibility" and "robustness in complex scenarios". Summary of the Invention

[0012] The purpose of this invention is to provide a method for recognizing left and right hands for multi-finger touchscreen interaction.

[0013] The objective of this invention can be achieved through the following technical solutions: A method for recognizing left and right hands for multi-finger touchscreen interaction includes: Step S1: Obtain touch data through the API interface, and identify the number of fingers and the type of action based on the touch data; Step S2: Rearrange all fingers and group all adjacent pairs of fingers into N-1 pairs of two fingers, where N is the number of fingers; Step S3: For each two-finger group, construct a spatial triangle based on the position and orientation of different fingers at the same time, and extract spatial features based on the spatial triangle; and construct a temporal triangle based on the position and orientation of the same finger at different times, and extract temporal features based on the temporal triangle. Step S4: Input the temporal and spatial features of each two-finger group into a first classifier with independent values ​​to obtain multiple first classification results corresponding one-to-one with the number of two-finger groups, wherein the first classification result represents the probability of left hand and / or right hand; Step S5: Merge the first classification results of all two-finger groups to obtain the left and right hand classification results.

[0014] The touch data obtained through the API interface in step S1 includes the position and orientation of the finger, as well as the corresponding timestamp.

[0015] Step S1 includes: Step S1-1: Segment the continuous touch data into multiple gesture segments based on the timestamp interval; Step S1-2: In each gesture segment, the number of fingers is obtained based on the number of touch points at the same time, the action type is obtained based on the average distance between touch points at different times, and the state sequence of each finger is obtained through distance constraints, timestamp alignment and outlier removal. Each element in the state sequence includes the position and orientation of the finger.

[0016] Step S2 includes: Step S2-1: Based on the state sequence of each finger, select the position and orientation of each finger at any given moment; Step S2-2: For each finger, draw a straight line through the position of the finger and its corresponding direction to obtain the first straight line corresponding to each finger; Step S2-3: Obtain the intersection points of all the first lines, and approximate a center point using the least squares circle fitting method; Step S2-4: Connect the position of each finger with the center point to obtain the first line segment corresponding to each finger; Step S2-5: Calculate the angle between each first line segment and the positive x-axis as the direction angle of each finger; Step S2-6: Sort all fingers according to their direction angle from smallest to largest; Step S2-7: Group all adjacent pairs of fingers together to obtain N-1 pairs of two fingers.

[0017] The construction of spatial triangles based on the positions and orientations of different fingers at the same moment, and the extraction of spatial features based on the spatial triangles, includes: Select any given moment and obtain the position and orientation of the two fingers at that moment; The positions of the two fingers are taken as the two vertices of the spatial triangle; For each of the two fingers, draw a straight line through the position of the finger and its corresponding direction to obtain the first straight line corresponding to each finger. Take the intersection of the first straight lines of the two fingers as the third vertex of the spatial triangle. The positional information of the three vertices of the spatial triangle is used as spatial features.

[0018] The construction of a time triangle based on the position and orientation of the same finger at different times, and the extraction of time features based on the time triangle, includes: Select any finger and obtain its position and orientation at two different times; The positions of the finger at the two moments are taken as the two vertices of the time triangle; At two points in time, draw a straight line through the position of the finger and its corresponding direction to obtain the first straight line corresponding to the finger at the two points in time. Take the intersection of the first straight lines at the two points in time as the third vertex of the time triangle. The positional information of the three vertices of the time triangle is used as the time feature.

[0019] In step S5, fusion is performed using the DS evidence theory.

[0020] In step S5, fusion is performed using a first fusion model, wherein the input of the first fusion model includes the action type and the first classification result of all two-finger groups.

[0021] A left and right hand recognition device for multi-finger touchscreen interaction includes a memory, a processor, and a program stored in the memory, wherein the processor executes the program to implement the method described above.

[0022] A storage medium having a program stored thereon, which, when executed, implements the method described above.

[0023] Compared with the prior art, the present invention has the following beneficial effects: 1. It only uses standard API data from standard touchscreens, namely PointerID, coordinates, and orientation angle, without requiring camera sensors, wearable devices, or underlying sensor interfaces. It is compatible with existing Android touch devices, reducing deployment costs and hardware barriers.

[0024] 2. By capturing general anatomical differences in the hand through "spatial and temporal triangle features," it eliminates the need for users to pre-enter samples (e.g., prompting users to operate with their left / right hand first), making it convenient and improving the user experience. Furthermore, due to the introduction of the temporal dimension, it is more robust to non-standard gestures (e.g., using the middle and ring fingers instead of the thumb and index finger), and it will not fail due to partial finger twisting or partial removal from the screen. Compared to other geometric methods, triangles have smaller geometric units than polygons, achieving greater benefits with lower computational cost. In addition, by rearranging and grouping fingers, it uniformly handles 2-finger to 5-finger gestures, and the model is scalable without the need for retraining.

[0025] 4. This invention also exhibits strong robustness in complex hand recognition scenarios involving non-standard gestures, maintaining good recognition accuracy. Attached Figure Description

[0026] Figure 1 This is a system architecture diagram of the present invention; Figure 2 This is a schematic diagram of the touch API data collected in this invention; Figure 3 This is a schematic diagram of a finger reordering scheme; Figure 4 A schematic diagram of the spatial triangle and the temporal triangle; Figure 5 A schematic diagram of the model design for the bi-finger classifier; Figure 6 This is a schematic diagram of a two-finger feature embedding scheme; Figure 7 This is a schematic diagram of the task framework for a multi-finger classifier; Figure 8 A diagram illustrating the HandiSense classification results; Figure 9 This is a schematic diagram of the main steps of the method of the present invention. Detailed Implementation

[0027] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0028] Most existing touchscreen hand recognition methods either rely on additional sensor devices, are not robust enough to complex scenarios, or depend on special permissions of APIs. These methods face key limitations in terms of scalability, usability, or versatility.

[0029] Raw touchscreen finger interaction data obtained directly from the local touch API may contain noisy readings and time discontinuities, which may corrupt the temporal characteristics of finger interactions. In addition, in multi-finger interaction scenarios, the data index assigned to each finger is unordered, potentially disrupting the spatial characteristics of the interaction data.

[0030] Compared to simple single-finger and two-finger interactions, hand recognition in multi-finger interactions requires consideration of interactions with varying numbers of fingers, making it a challenging and complex task. Existing solutions primarily design features for standard finger gestures (such as using the thumb and index finger to perform magnification tasks) and build finger recognition schemes accordingly. However, in real-world scenarios, users may use non-standard finger combinations (such as using the middle and ring fingers instead of the thumb and index finger to perform magnification gestures). The spatial layout and motion trajectory of these non-standard gestures deviate significantly from standard gestures, leading to a substantial decrease in recognition accuracy.

[0031] This application provides a method for recognizing left and right hands in multi-finger touchscreen interaction, such as... Figure 1 and Figure 9 As shown, it includes: Step S1: Obtain touch data through the API interface, and identify the number of fingers and the type of action based on the touch data. The touch data obtained through the API interface includes the position and orientation of the fingers, as well as the corresponding timestamp.

[0032] In this embodiment, capacitive touch data sampled by the native API is collected from Android touchscreen devices, such as... Figure 2 As shown, it includes PointerID, the center coordinates x and y of the touch area, and the orientation θ. The orientation is an angle value. Specifically, since the touch is not a point, but generally an elliptical area, the orientation θ is only the direction angle of the major axis of the elliptical area.

[0033] Step S1 includes: Step S1-1: Segment the continuous touch data into multiple gesture segments based on the timestamp interval; Step S1-2: In each gesture segment, the number of fingers is obtained based on the number of touch points at the same time, the action type is obtained based on the average distance between touch points at different times, and the state sequence of each finger is obtained through distance constraints, timestamp alignment and outlier removal. Each element in the state sequence includes the position and orientation of the finger.

[0034] For each gesture lasting K frames using N fingers, N time-aligned PointerID sequences are recorded, clipped at motion boundaries, and the resulting segments are labeled. This application focuses on three core interaction primitives—zoom in, zoom out, and swipe—performed using four-finger configurations (two, three, four, and five fingers), providing a total of 12 gesture classes. However, users can perform gestures using optional finger combinations that still satisfy the finger counting and advanced gesture category criteria (e.g., using the middle and ring fingers instead of the thumb and index finger for two-finger zoom). These non-standard gestures were also collected to capture the natural variability of human interaction styles. Therefore, each gesture class is divided into two execution styles: standard gestures and non-standard gestures. Table 1 shows the settings of the gesture dataset, including different finger settings, gesture types, and execution styles.

[0035] Table 1. Dataset configuration for the target gesture Step S2: Rearrange all fingers and group all adjacent pairs of fingers into N-1 pairs, where N is the number of fingers, including: Step S2-1: Based on the state sequence of each finger, select the position and orientation of each finger at any given moment; Step S2-2: For each finger, draw a straight line through the position of the finger and its corresponding direction to obtain the first straight line corresponding to each finger; Step S2-3: Obtain the intersection points of all the first lines, and approximate a center point using the least squares circle fitting method; Step S2-4: Connect the position of each finger with the center point to obtain the first line segment corresponding to each finger; Step S2-5: Calculate the angle between each first line segment and the positive x-axis as the direction angle of each finger; Step S2-6: Sort all fingers according to their direction angle from smallest to largest; Step S2-7: Group all adjacent pairs of fingers together to obtain N-1 pairs of two fingers.

[0036] This application designs a data preprocessing pipeline designed to align, filter, and reorder data from multiple operating fingers. Specifically, the system first segments the continuous raw data into gesture-level time windows using timestamp intervals. Within each gesture segment, time alignment is performed by ensuring frame-by-frame synchronization of all active fingers. Outlier removal is also applied to remove frames with sudden changes in finger speed or missing position records. These steps produce a reasonable and temporally stable set of multi-finger data. A finger reordering scheme based on relative angle positioning is then designed, first retrieving the real-time coordinates and orientation parameters of each finger from the touch input data. Next, the pairwise intersections of the direction vectors of each finger are calculated: for each pair of fingers, their respective direction vectors (derived from the direction data) are expanded until they intersect, generating a set of potential intersection coordinates. Then, a center point is approximated using a least-squares circle fitting method. This fitting process minimizes the sum of squared distances between the remaining intersections and the assumed circle, ultimately obtaining the center coordinates (x0, y0), as shown below. Figure 3 As shown in the left figure. Then, with this center point as the origin, the angle between the positive horizontal axis and the straight line from the origin to the finger contact point is calculated. The angle calculation follows standard trigonometric methods, using the arctangent of the ratio of the vertical distance from the contact point to the horizontal distance, and is adjusted for quadrant positioning to ensure accuracy. Each finger is assigned a consistent label (AE) by sorting them according to their angular position relative to the fitted center (from 0° to 360°), such as... Figure 3 As shown in the right figure. For example, an angle α greater than an angle β indicates that point B is after point A. This label remains stable in repeated gestures, resolving inconsistencies caused by the native touch API. Through the above preprocessing steps, the spatiotemporal features are refined at the data level, laying a unified foundation for further processing.

[0037] Step S3: For each two-finger group, construct a spatial triangle based on the position and orientation of different fingers at the same time, and extract spatial features based on the spatial triangle; and construct a temporal triangle based on the position and orientation of the same finger at different times, and extract temporal features based on the temporal triangle. Specifically, for spatial features, spatial triangles are constructed based on the positions and orientations of different fingers at the same time, and spatial features are extracted based on these spatial triangles, including: Select any given moment and obtain the position and orientation of the two fingers at that moment; The positions of the two fingers are taken as the two vertices of the spatial triangle; For each of the two fingers, draw a straight line through the position of the finger and its corresponding direction to obtain the first straight line corresponding to each finger. Take the intersection of the first straight lines of the two fingers as the third vertex of the spatial triangle. The positional information of the three vertices of the spatial triangle is used as spatial features.

[0038] In this embodiment, a time triangle is constructed based on the position and orientation of the same finger at different times, and time features are extracted based on the time triangle, including: Select any finger and obtain its position and orientation at two different times; The positions of the finger at the two moments are taken as the two vertices of the time triangle; At two points in time, draw a straight line through the position of the finger and its corresponding direction to obtain the first straight line corresponding to the finger at the two points in time. Take the intersection of the first straight lines at the two points in time as the third vertex of the time triangle. The positional information of the three vertices of the time triangle is used as the time feature.

[0039] Spatial and temporal triangulation features are constructed to jointly model the geometric layout and motion trajectory of the fingers. Specifically, spatial triangulation features are constructed by calculating the spatial relationship between two fingers within the same time frame. Figure 4 As shown in (A), assuming two touching fingers are in the same time frame. t Location information, represented as , and , and their locations and ,in and It is the angle with the positive horizontal axis. Based on this joking information, the equation of the straight line corresponding to each contact point is derived: Rearrange to standard format ,get: Therefore, static intersection point ( , The coordinates of ) are: Spatial triangle features are formed based on the coordinates of three points in each time frame. The side lengths, angles, and spatial coordinates of the triangles describe in detail the spatial layout of the user's two-finger interaction. Since the left and right hands have different spatial shapes, these features can directly help the operator recognize the task.

[0040] To further consider capturing the motion trajectory of a single finger across consecutive time frames, it is represented using a temporal triangle feature. For example... Figure 4 As shown in (B), the trajectory of the finger across consecutive time frames can be represented as follows: For each time frame There is a contact point with coordinates as follows: , and direction information Its data structure and calculation formula are similar to spatial triangle features, and it can construct up to n−1 consecutive temporal triangles. Compared with spatial triangle features that capture the spatial relationship between two fingers in the same time frame, temporal triangle features utilize the trajectory differences formed by each finger in consecutive time frames. These features contain rich information on finger motion and temporal dynamics, further supporting the recognition task of two-finger manipulators.

[0041] Step S4: Input the temporal and spatial features of each two-finger group into a first classifier with independent values ​​to obtain multiple first classification results corresponding one-to-one with the number of two-finger groups. The first classification result represents the probability of left hand and / or right hand. This application constructs a classification model for identifying the operator hand in a two-finger subtask, and its design is as follows: Figure 5 As shown, specifically, two parallel classification pipelines are constructed for temporal and spatial triangle features, respectively. In each pipeline, the input triangle features are encoded into fixed-dimensional vectors. These vectors are then modeled through multiple TransformerEncode layers, and the final classification head outputs the operator probability. The pipeline utilizes an attention mechanism to focus attention on keyframes, dynamically assigning weights to various keyframes. Based on Dempster-Shafer (DS) evidence theory, the model additionally designs a conflict-conditional loss function to optimize the classification results, integrating the results from the temporal and spatial feature classification pipelines to provide a more robust and reliable classification output. Specifically, it is assumed that both temporal and spatial triangle features contribute to operator recognition, and each feature provides a probability of identifying the operator. The final model probability fusion can be represented as follows: in The closer K is to 1, the greater the conflict between the two classification pipelines. This can be interpreted as a measure of mutual information; therefore, K is used as the baseline for the conflict-conditional loss function. The model's final output is a unified classification decision that labels the interaction as left-handed or right-handed.

[0042] In two-finger hand recognition tasks, temporal and spatial triangular features exhibit stable performance for gestures that produce different triangular features (e.g., zooming in and out). However, they lack sufficient stability when processing gestures without highly diverse triangular features (e.g., swiping). To address this issue, the HandiSense system further proposes a two-finger feature embedding strategy, such as... Figure 6As shown. During the training of the two-finger classification model, features are first extracted from the zoom-in / zoom-out gesture. These extracted features are then concatenated to form the final feature representation. The extracted features are treated as query values, while the feature vector encoded by the swipe gesture is used as a key-value pair. The embedding process can be represented as: in, This represents the dot product of the query and the key, which measures the relevance between them. This strategy allows the model to dynamically adjust weights and features of interest during the recognition process, thus adapting more flexibly to different user interactions.

[0043] Step S5: Merge the first classification results of all two-finger groups to obtain the left and right hand classification results. In some embodiments, the fusion is performed using the DS evidence theory.

[0044] In this embodiment, fusion is performed through a first fusion model, wherein the input of the first fusion model includes the action type and the first classification result of all two-finger groups.

[0045] Based on the constructed sub-task model, the HandiSense system further builds a scalable multi-finger interaction feature framework. This framework unifies the modeling of different numbers of fingers and supports hand recognition tasks across arbitrary multi-finger gestures. The overall task framework is as follows: Figure 7 As shown, after data preprocessing, adjacent finger pairs are systematically identified. Then, for each adjacent pair, temporal and spatial triangular features are constructed to capture the instantaneous spatial structure and temporal evolution of the interaction. A modular multi-branch classification architecture is then designed, where each bifurcation feature unit is processed independently by a dedicated sub-classifier. Based on the number of detected fingers, the system selectively activates relevant sub-task branches and applies a controlled masking mechanism. Specifically, a modular multi-branch classification structure is designed where each adjacent pair of fingers (i.e., AB, BC, CD, DE) corresponds to an independent two-finger classifier branch. All branches share the same architecture and parameters, enabling efficient parameter reuse and consistent feature processing under different input conditions. The system dynamically activates relevant branches based on the number of detected fingers; for example, in two-finger interactions, only the AB branch participates, while in three-finger interactions, both the AB and BC branches are utilized. Then, before final prediction, a selective fusion mechanism is introduced in the multi-branch bifurcation classifier module to dynamically adapt to different finger configurations. For each input sample, adjacent bifurcation tasks are activated or masked based on the number of available fingers and a predefined masking probability. This selective fusion strategy enables the system to adaptively adjust to different interaction complexities without retraining, ensuring robust and consistent operator hand classification across different finger configurations.

[0046] To verify the effectiveness of the proposed HandiSense system, this application implemented and evaluated it on a capacitive multi-touch screen platform with a sampling frequency of 120 Hz. All data processing and model training were performed on a workstation equipped with a PC-based platform. This application utilized multi-finger gesture samples from 40 participants, covering three gesture types (zoom in, zoom out, and swipe) and four finger configurations (two to five fingers), with non-standard gestures belonging to one of these three types. In some experiments, the sample data for non-standard gestures needed to be separated independently.

[0047] To assess generalization ability under real-world usage conditions, participants performed gestures in different postures (sitting, standing, and walking) on ​​four types of devices: two tablets and two smartphones. Each gesture was performed with both the left and right hands, all samples were temporally consistent, and labeled by self-report and observer validation. This application segments the raw input into gesture-level sequences of 20–60 frames and processes them into static and dynamic triangular features.

[0048] Figure 8 The classification accuracy for left and right hands was demonstrated for different multi-finger gesture types, including two-finger (2F), three-finger (3F), four-finger (4F), and five-finger (5F) interactions. Overall, the system achieved a consistently high accuracy of over 96% across all gestures, validating the effectiveness and robustness of HandiSense.

[0049] The proposed model is based on the Transformer framework and was compared with four representative baseline models. To ensure a fair comparison, this application presents two case studies. In Case Study 1, the baseline model used raw data from the touch API as training data (represented as Raw in Table 2). In Case Study 2, this application provided features extracted from the spatiotemporal feature extraction module to train the baseline model (represented as Features in Table 2). The results are shown in Table 2.

[0050] Table 2. Accuracy of baseline model settings for this dataset Under interaction conditions ranging from two to five fingers, the method of this invention consistently outperforms all baseline models in both cases (original and featured). Specifically, for the two-finger setting, our model achieves significantly higher accuracy than the second-best performing MLP or LSTM. These results validate the effectiveness of the Transformer framework of this application for various gesture configurations and gesture types.

[0051] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for recognizing left and right hands in multi-finger touchscreen interaction, characterized in that, include: Step S1: Obtain touch data through the API interface, and identify the number of fingers and the type of action based on the touch data; Step S2: Rearrange all fingers and group all adjacent pairs of fingers into N-1 pairs of two fingers, where N is the number of fingers; Step S3: For each two-finger group, construct a spatial triangle based on the position and orientation of different fingers at the same time, and extract spatial features based on the spatial triangle; and construct a temporal triangle based on the position and orientation of the same finger at different times, and extract temporal features based on the temporal triangle. Step S4: Input the temporal and spatial features of each two-finger group into a first classifier with independent values ​​to obtain multiple first classification results corresponding one-to-one with the number of two-finger groups, wherein the first classification result represents the probability of left hand and / or right hand; Step S5: Merge the first classification results of all two-finger groups to obtain the left and right hand classification results.

2. The method for recognizing left and right hands in multi-finger touchscreen interaction according to claim 1, characterized in that, The touch data obtained through the API interface in step S1 includes the position and orientation of the finger, as well as the corresponding timestamp.

3. The method for recognizing left and right hands in multi-finger touchscreen interaction according to claim 2, characterized in that, Step S1 includes: Step S1-1: Segment the continuous touch data into multiple gesture segments based on the timestamp interval; Step S1-2: In each gesture segment, the number of fingers is obtained based on the number of touch points at the same time, the action type is obtained based on the average distance between touch points at different times, and the state sequence of each finger is obtained through distance constraints, timestamp alignment and outlier removal. Each element in the state sequence includes the position and orientation of the finger.

4. The method for recognizing left and right hands in multi-finger touchscreen interaction according to claim 3, characterized in that, Step S2 includes: Step S2-1: Based on the state sequence of each finger, select the position and orientation of each finger at any given moment; Step S2-2: For each finger, draw a straight line through the position of the finger and its corresponding direction to obtain the first straight line corresponding to each finger; Step S2-3: Obtain the intersection points of all the first lines, and approximate a center point using the least squares circle fitting method; Step S2-4: Connect the position of each finger with the center point to obtain the first line segment corresponding to each finger; Step S2-5: Calculate the angle between each first line segment and the positive x-axis as the direction angle of each finger; Step S2-6: Sort all fingers according to their direction angle from smallest to largest; Step S2-7: Group all adjacent pairs of fingers together to obtain N-1 pairs of two fingers.

5. The method for recognizing left and right hands in multi-finger touchscreen interaction according to claim 1, characterized in that, The construction of spatial triangles based on the positions and orientations of different fingers at the same moment, and the extraction of spatial features based on the spatial triangles, includes: Select any given moment and obtain the position and orientation of the two fingers at that moment; The positions of the two fingers are taken as the two vertices of the spatial triangle; For each of the two fingers, draw a straight line through the position of the finger and its corresponding direction to obtain the first straight line corresponding to each finger. Take the intersection of the first straight lines of the two fingers as the third vertex of the spatial triangle. The positional information of the three vertices of the spatial triangle is used as spatial features.

6. The method for recognizing left and right hands for multi-finger touchscreen interaction according to claim 1, characterized in that, The construction of a time triangle based on the position and orientation of the same finger at different times, and the extraction of time features based on the time triangle, includes: Select any finger and obtain its position and orientation at two different times; The positions of the finger at the two moments are taken as the two vertices of the time triangle; At two points in time, draw a straight line through the position of the finger and its corresponding direction to obtain the first straight line corresponding to the finger at the two points in time. Take the intersection of the first straight lines at the two points in time as the third vertex of the time triangle. The positional information of the three vertices of the time triangle is used as the time feature.

7. The method for recognizing left and right hands in multi-finger touchscreen interaction according to claim 1, characterized in that, In step S5, fusion is performed using the DS evidence theory.

8. The method for recognizing left and right hands in multi-finger touchscreen interaction according to claim 1, characterized in that, In step S5, fusion is performed using a first fusion model, wherein the input of the first fusion model includes the action type and the first classification result of all two-finger groups.

9. A left / right hand recognition device for multi-finger touchscreen interaction, comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-8.

10. A storage medium having a program stored thereon, characterized in that, When the program is executed, it implements the method as described in any one of claims 1-8.

Images