Method and apparatus for finger pose estimation based on touch screen

Abstract

The application discloses a finger posture estimation method and device based on a touch screen, and the method comprises the following steps: acquiring a touch screen image of a user's finger; inputting the touch screen image into a pre-trained time sequence finger posture estimation model, and outputting a finger posture corresponding to the touch screen image, wherein the time sequence finger posture estimation model comprises an image feature extraction network, a time sequence feature fusion network and a finger posture prediction network. The application trains the time sequence finger posture estimation model based on continuous touch screen images of the finger to predict the finger posture, so that the prediction accuracy of the finger posture can be improved, and the input mode of human-computer interaction can be enriched.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction, and in particular to a method and apparatus for estimating finger pose based on a touchscreen. Background Technology

[0002] Touchscreens have become the human-computer interaction input device for a large number of smart mobile devices due to their simplicity, speed, and low cost. However, current touchscreens only report the planar position of a finger touching the screen. Although human fingers are very flexible, the degree of freedom that touchscreens offer to a single finger is quite limited. When dealing with complex interactive applications, multiple fingers are often needed to complete the operation. The introduction of multi-finger gestures increases the learning cost and difficulty of using touchscreen devices.

[0003] If touchscreens could provide more input information about a single finger, their expressive power would be enhanced, application interaction logic would be simplified, and more human-computer interaction input modes would be introduced. One feasible approach is to estimate the angle of a finger touch from the touchscreen image. As the finger posture changes, the touchscreen image formed by pressing also changes. Inferring the finger posture from the touch image would provide upper-layer applications with additional degrees of posture freedom on top of the original touch position.

[0004] Existing technologies for estimating finger postures from touchscreen devices can be divided into two categories based on whether they rely on additional devices:

[0005] (1) Solutions requiring additional equipment

[0006] Some researchers attach a camera to a user's fingertip and calculate pitch and offset angles by detecting changes in light intensity on the fingernail. Others use depth-sensing cameras to acquire fingerprint depth information and constrain the pitch angle range to 0–90 degrees by fusing prior knowledge. Still others fix a depth camera to a tablet to create a 3D point cloud of the finger and match it with a standard finger model to obtain the current finger pose.

[0007] However, the above solution requires additional sensor assistance, making it difficult to promote in practical applications.

[0008] (2) Solution that does not require additional equipment

[0009] Some researchers determine finger deflection angles based on the shape of touchscreen images. Others use contact area, average grayscale, and shape symmetry as input features to train a deep neural network to estimate finger deflection and pitch angles. Still others have expanded this to 21 manually defined features and trained a Gaussian process regressor to predict finger pose. Still others have improved the accuracy of finger pose prediction by collecting large-scale datasets of touchscreen images and their corresponding poses and training deep convolutional neural networks.

[0010] Although the above solutions directly utilize touchscreen images as input to the algorithm, eliminating the need for additional sensors, they all use single-frame touchscreen images as input and fail to effectively utilize continuous information about finger posture changes. Therefore, in practical applications, this can lead to significant jitter and inaccuracies, resulting in a degraded user experience. Furthermore, while some existing methods utilize machine learning sampling and construct large-scale datasets, individual differences in finger posture remain a significant challenge. Summary of the Invention

[0011] This application aims to at least partially address one of the technical problems in the related art.

[0012] Therefore, the purpose of this application is to improve the accuracy of finger pose prediction and reduce the impact of individual differences in fingers on finger pose prediction. A finger pose estimation method based on touch screen is proposed.

[0013] Another objective of this application is to propose a finger pose estimation device based on a touch screen.

[0014] To achieve the above objectives, this application proposes a finger pose estimation method based on a touchscreen, including:

[0015] Acquire an image of the user's finger on the touchscreen;

[0016] The touchscreen image is input into a pre-trained temporal finger pose estimation model, which outputs the finger pose corresponding to the touchscreen image. The temporal finger pose estimation model includes an image feature extraction network, a temporal feature fusion network, and a finger pose prediction network.

[0017] In one possible implementation, before inputting the touchscreen image into a pre-trained temporal finger pose estimation model and outputting the finger pose corresponding to the touchscreen image, the following steps are included:

[0018] Obtain a training set of touchscreen images of at least one training user's fingers and a training set of finger postures corresponding to the training set of touchscreen images;

[0019] Train a general temporal model to be trained based on the touch screen image training set and the finger posture training set, and obtain the trained general temporal model;

[0020] Obtain a specific set of touchscreen images of a specific user's finger and a specific set of finger poses corresponding to the specific set of touchscreen images;

[0021] The general temporal model is fine-tuned based on the specific touchscreen image training set and the specific finger pose training set to obtain the pre-trained temporal finger pose estimation model.

[0022] In one possible implementation, the step of training a general temporal model to be trained based on the touchscreen image training set and the finger pose training set, and obtaining the trained general temporal model, includes:

[0023] The touchscreen image training set is input into the single-frame image feature extraction module, and the single-frame image features are output.

[0024] The single-frame image features are input into the temporal feature fusion module, which outputs image sequence features.

[0025] The image sequence features are input into the finger pose prediction module, which outputs the finger pose sequence.

[0026] Calculate the error between the finger sequence pose and the finger pose training set;

[0027] Based on the error, the time series model to be trained is trained using the gradient descent algorithm to obtain the trained general time series model.

[0028] In one possible implementation, obtaining a specific touchscreen image training set for a specific user's finger and a specific finger pose training set corresponding to the specific touchscreen image training set includes:

[0029] By instructing the specific user to lift their finger from a flat position to an upright position in the direction of the arrow displayed on the interface, a specific touchscreen image training set and a specific finger posture training set are obtained.

[0030] In one possible implementation, calculating the error between the finger sequence pose and the finger pose training set includes:

[0031] The error between the finger sequence pose and the finger pose training set is calculated using a preset loss function, wherein the formula for the preset loss function is as follows:

[0032]

[0033] Where, α kThis represents the training value of the deflection angle corresponding to the k-th frame image. β represents the predicted deflection angle corresponding to the k-th frame image. k This represents the training value of the pitch angle corresponding to the k-th frame image. L represents the predicted pitch angle value corresponding to the k-th frame image. MSE Let N represent the mean squared error loss function, and let N represent the sequence length of the touchscreen image training set.

[0034] To achieve the above objectives, this application also proposes a finger pose estimation device based on a touchscreen, comprising:

[0035] The first acquisition module is used to acquire the touch screen image of the user's finger;

[0036] The output module is used to input the touch screen image into a pre-trained temporal finger pose estimation model and output the finger pose corresponding to the touch screen image. The temporal finger pose estimation model includes an image feature extraction network, a temporal feature fusion network, and a finger pose prediction network.

[0037] In one possible implementation, the device further includes:

[0038] The second acquisition module is used to acquire a touch screen image training set of at least one training user's finger and a finger posture training set corresponding to the touch screen image training set;

[0039] The first training module is used to train a general temporal model to be trained based on the touch screen image training set and the finger posture training set, and to obtain the trained general temporal model.

[0040] The third acquisition module is used to acquire a specific touch screen image training set of a specific user's finger and a specific finger posture training set corresponding to the specific touch screen image training set;

[0041] The second training module is used to fine-tune the general temporal model based on the specific touch screen image training set and the specific finger posture training set, and obtain the pre-trained temporal finger posture estimation model.

[0042] In one possible implementation, the first training module includes:

[0043] The first output unit is used to input the touch screen image training set into the single frame image feature extraction module and output the single frame image features.

[0044] The second output unit is used to input the single-frame image features into the temporal feature fusion module and output image sequence features;

[0045] The third output unit is used to input the image sequence features into the finger pose prediction module and output the finger sequence pose.

[0046] A computing unit is used to calculate the error between the finger sequence pose and the finger pose training set;

[0047] The training unit is used to train the time-series model to be trained using a gradient descent algorithm based on the error, and obtain the trained general time-series model.

[0048] In one possible implementation, the third acquisition module includes:

[0049] The acquisition unit is used to acquire the specific touch screen image training set and the specific finger posture training set by instructing the specific user to lift their finger from a flat state to an upright state in the direction of the arrow displayed on the interface.

[0050] In one possible implementation, the computing unit includes:

[0051] The computational subunit is used to calculate the error between the finger sequence pose and the finger pose training set using a preset loss function, wherein the formula for the preset loss function is as follows:

[0052]

[0053] Where, α k This represents the training value of the deflection angle corresponding to the k-th frame image. β represents the predicted deflection angle corresponding to the k-th frame image. k This represents the training value of the pitch angle corresponding to the k-th frame image. L represents the predicted pitch angle value corresponding to the k-th frame image. MSE Let N represent the mean squared error loss function, and let N represent the sequence length of the touchscreen image training set.

[0054] The beneficial effects of this application are:

[0055] In this embodiment, a touchscreen image of a user's finger is acquired; the touchscreen image is input into a pre-trained temporal finger pose estimation model, which outputs the finger pose corresponding to the touchscreen image. The temporal finger pose estimation model includes an image feature extraction network, a temporal feature fusion network, and a finger pose prediction network. This application trains the temporal finger pose estimation model based on continuous touchscreen images of the finger to predict finger poses, which can improve the accuracy of finger pose prediction and enrich human-computer interaction input modes.

[0056] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0057] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0058] Figure 1 This is a flowchart of a touchscreen-based finger pose estimation method according to an embodiment of this application;

[0059] Figure 2 A schematic diagram illustrating the three-dimensional posture definition of a finger when pressing a touchscreen in the prior art;

[0060] Figure 3 This is a schematic diagram of a finger posture acquisition method based on optical tracking technology in the prior art;

[0061] Figure 4 This is a schematic diagram illustrating the training process of a temporal finger pose estimation model according to an embodiment of this application;

[0062] Figure 5 This is a schematic diagram illustrating the training process of a general time series model according to an embodiment of this application;

[0063] Figure 6 This is a schematic diagram of the interface of a specific user finger posture acquisition software according to an embodiment of this application;

[0064] Figure 7 This is a schematic diagram of the structure of a touchscreen-based finger posture estimation device according to an embodiment of this application. Detailed Implementation

[0065] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0066] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0067] The following describes, with reference to the accompanying drawings, a finger pose estimation method and apparatus based on a touchscreen according to embodiments of this application. First, the finger pose estimation method based on a touchscreen according to embodiments of this application will be described with reference to the accompanying drawings.

[0068] Figure 1This is a flowchart of a touchscreen-based finger pose estimation method according to an embodiment of this application.

[0069] like Figure 1 As shown, the touchscreen-based finger pose estimation method includes:

[0070] Step S110: Obtain the touchscreen image of the user's finger.

[0071] In this embodiment of the application, when a user operates the device by touching the touch screen with their finger, an image of the user's finger on the touch screen can be acquired.

[0072] It should be noted that there are several ways to obtain the touchscreen image of a user's finger, including but not limited to the following two: First, modify the device's touchscreen driver so that when the user operates the device by touching the touchscreen with their finger, the device outputs the original capacitance matrix to obtain the touchscreen image of the user's finger; Second, connect a separate capacitive touchscreen and its controller to a computer and use a special debugging program to obtain its capacitance matrix, thereby obtaining the touchscreen image of the user's finger.

[0073] Step S120: Input the touch screen image into the pre-trained temporal finger pose estimation model and output the finger pose corresponding to the touch screen image. The temporal finger pose estimation model includes an image feature extraction network, a temporal feature fusion network, and a finger pose prediction network.

[0074] In this embodiment, after acquiring a touchscreen image of a user's finger, the touchscreen image can be input into a pre-trained temporal finger pose estimation model, which outputs the finger pose corresponding to the touchscreen image. The temporal finger pose estimation model includes an image feature extraction network, a temporal feature fusion network, and a finger pose prediction network. That is, considering the temporal features of the touchscreen image, the temporal finger pose estimation model, including the image feature extraction network, the temporal feature fusion network, and the finger pose prediction network, is pre-trained to obtain the pre-trained temporal finger pose estimation model. Then, the touchscreen image is input into the pre-trained temporal finger pose estimation model, which outputs the finger pose corresponding to the touchscreen image.

[0075] It should be noted that, Figure 2 This is a schematic diagram illustrating the three-dimensional posture definition of a finger when pressing a touchscreen in existing technology, such as... Figure 2 As shown, when a finger presses the touch screen, the finger posture can include yaw angle, pitch angle and scroll angle. This application embodiment only predicts the finger posture of yaw angle and pitch angle.

[0076] It should be noted that there are multiple ways to acquire finger posture data. For example, a three-axis gyroscope can be attached to both the user's finger and the touchscreen. By measuring the gyroscope readings on both the finger and the touchscreen, the finger posture can be determined based on the difference between these readings. Alternatively, optical tracking technology can be used. Figure 3 This is a schematic diagram of a prior art finger pose acquisition method based on optical tracking technology, such as... Figure 3 As shown, several light-sensitive balls can be attached to a finger and a touchscreen, and the finger posture can be obtained by calculating the transformation matrix between the finger and touchscreen coordinate systems. Furthermore, the finger can be supported by molds at various tilt angles to obtain images from different angles, thereby acquiring the finger posture. Therefore, in this embodiment, the method for obtaining the finger posture is not specifically limited.

[0077] In this embodiment, a touchscreen image of a user's finger is acquired; the touchscreen image is input into a pre-trained temporal finger pose estimation model, which outputs the finger pose corresponding to the touchscreen image. The temporal finger pose estimation model includes an image feature extraction network, a temporal feature fusion network, and a finger pose prediction network. This application trains the temporal finger pose estimation model based on continuous touchscreen images of the finger to predict finger poses, which can improve the accuracy of finger pose prediction and enrich human-computer interaction input modes.

[0078] In one possible implementation, before inputting the touchscreen image into a pre-trained temporal finger pose estimation model and outputting the finger pose corresponding to the touchscreen image, the following steps are included:

[0079] Obtain a training set of touchscreen images of at least one training user's finger and a training set of finger poses corresponding to the touchscreen image training set;

[0080] Train a general temporal model to be trained based on the touch screen image training set and the finger posture training set, and obtain the trained general temporal model;

[0081] Obtain a specific set of touchscreen images of a specific user's finger and a specific set of finger poses corresponding to the specific set of touchscreen images;

[0082] Fine-tune the general temporal model based on a specific touchscreen image training set and a specific finger pose training set to obtain a pre-trained temporal finger pose estimation model.

[0083] In this embodiment, considering the pre-training of the temporal finger pose estimation model, a training set of touchscreen images of at least one training user's finger and a corresponding finger pose training set can be obtained. Then, a general temporal model can be trained based on the touchscreen image training set and the finger pose training set to obtain the trained general temporal model. Next, a specific touchscreen image training set for a specific user's finger and a corresponding specific finger pose training set can be obtained. Finally, the general temporal model can be fine-tuned based on the specific touchscreen image training set and the specific finger pose training set to obtain the pre-trained temporal finger pose estimation model. The general temporal model can be a general version of the temporal finger pose estimation model. Considering the differences in finger pressure on the touchscreen among different users, including force habits, finger pose distribution, and finger shape, the general temporal model's prediction effect on different users' finger poses is poor. Therefore, a general temporal model can be trained for a specific user to obtain the temporal finger pose estimation model, thereby improving the finger pose prediction effect of the temporal finger pose estimation model. For example, Figure 4 This is a schematic diagram illustrating the training process of a temporal finger pose estimation model according to an embodiment of this application, as shown below. Figure 4 As shown, temporal touchscreen images (a training set of touchscreen images for at least one training user's finger) and ground truth finger poses (a training set of finger poses corresponding to the touchscreen image training set) can be acquired. Then, a general machine learning model (a general temporal model) is trained based on the temporal touchscreen images and the ground truth finger poses. Additionally, user data can be collected (a specific set of touchscreen images for a specific user's finger and a specific set of finger poses corresponding to that specific set of touchscreen images can be acquired). Based on this user data, the general machine learning model is fine-tuned, ultimately yielding a user machine learning model (a temporal finger pose estimation model). Thus, through these two rounds of training, a pre-trained temporal finger pose estimation model can be obtained. Furthermore, using this model to predict user finger poses can improve the accuracy of finger pose prediction and reduce the impact of individual finger differences on finger pose prediction.

[0084] It should be noted that the general temporal model is fine-tuned based on specific touchscreen image training sets and specific finger pose training sets. The fine-tuning methods include, but are not limited to, fine-tune, linear-probe, and few-shot learning methods.

[0085] In one possible implementation, a general temporal model to be trained is trained based on a touchscreen image training set and a finger pose training set, and the trained general temporal model is obtained, including:

[0086] The touchscreen image training set is input into the single-frame image feature extraction module, and the single-frame image features are output.

[0087] The features of a single frame image are input into the temporal feature fusion module, which outputs image sequence features.

[0088] The image sequence features are input into the finger pose prediction module, which outputs the finger pose sequence.

[0089] Calculate the error between the finger sequence pose and the finger pose training set;

[0090] Based on the error, the time series model to be trained is trained using the gradient descent algorithm to obtain the trained general time series model.

[0091] In this embodiment, the general temporal model may include a single-frame image feature extraction module, a temporal feature fusion module, and a finger pose prediction module. The single-frame image feature extraction module may include convolutional layers, pooling layers, and activation layers. The input to the single-frame image feature extraction module can be a single-frame image, and the output can be a fixed-dimensional feature descriptor, i.e., single-frame image features. A typical structure for the single-frame image feature extraction module is a convolutional neural network. The temporal feature fusion module can integrate and map the aforementioned single-frame images to form features describing the entire touchscreen image sequence, i.e., image sequence features. A typical structure for the temporal feature fusion module may include an attention network and a long short-term memory network. The input to the finger pose prediction module can be the aforementioned image sequence features, and the output can be finger pose sequences. A typical structure for the finger pose prediction module may include fully connected layers and nonlinear mapping layers. Specifically, Figure 5 This is a schematic diagram illustrating the training process of a general time-series model according to embodiments of this application, such as... Figure 5 As shown, the training process of the general temporal model can include: inputting the touchscreen image training set (touchscreen image sequence) into the single-frame image feature extraction module, outputting single-frame image features; then inputting the single-frame image features into the temporal feature fusion module, outputting image sequence features; then inputting the image sequence features into the finger pose prediction module, outputting finger sequence poses; finally, calculating the error between the finger sequence poses and the finger pose training set, and training the temporal model to be trained using the gradient descent algorithm based on the error, thus obtaining the trained general temporal model. In this way, by training the single-frame image feature extraction module, the temporal feature fusion module, and the finger pose prediction module, a trained general temporal model can be obtained, enabling the prediction of finger poses using the temporal image sequence features of touchscreen images, thereby improving the accuracy of finger pose prediction.

[0092] It is understandable that the single-frame image feature extraction module and the temporal feature fusion module can be merged into one module, that is, after inputting the touch screen image training set, the image sequence features are directly output.

[0093] In one possible implementation, obtaining a specific touchscreen image training set for a specific user's finger and a specific finger pose training set corresponding to the specific touchscreen image training set includes:

[0094] By instructing specific users to lift their fingers from a flat position to an upright position in the direction of the arrows displayed on the interface, a specific set of touch screen images and a specific set of finger postures are obtained.

[0095] In this embodiment, a specific touchscreen image training set and a specific finger posture training set can be obtained by instructing a specific user to lift their finger from a flat position to an upright position according to the arrow direction displayed on the interface. For example, an application software specifically designed to record touchscreen images and the corresponding finger postures can be developed. Figure 6 This is a schematic diagram of the interface of a specific user finger gesture acquisition software according to an embodiment of this application, such as... Figure 6 As shown, the software interface displays an arrow indicating the current registration deflection angle, the current registration progress, and the XY position change curve over time t after registration. After the user clicks "Start," the interface displays an arrow indicating the current registration deflection angle. Following the application's prompts, the user evenly lifts their finger from a flat position (0°) to an upright position (90°) and then removes it from the screen. Understandably, the user can also rotate their finger from an upright to a flat position according to the arrow direction, or rotate their finger from the initial state to the final state according to any initial and final arrow direction, and repeat this process multiple times. Since the finger lifting process involves a change in contact position, the linearity of the contact position (X or Y) change over time t can be used to determine the linearity of the pitch angle change during the finger lifting process. If the curve is highly non-linear, the user will be prompted to re-register. It should be noted that excessively long or short lifting times can easily cause jitter during the lifting process, resulting in registration failure and prompting the user to re-register.

[0096] Understandably, once the registration process is successfully completed, a sequence of touchscreen images of length N will be obtained. The direction of the indicator arrow will be used as the pseudo-true value for the deflection angle, while the N values ​​obtained by uniform interpolation within the range of (0° to 90°) will be used as the pseudo-true values ​​for the pitch angle. After appropriate user guidance and learning, users will be able to quickly complete the recording of finger lift sequences at various angles. The resulting touchscreen images and their corresponding finger posture pseudo-true values ​​will be stored as a specific dataset for that finger.

[0097] In this way, a specific set of touch screen images and a specific set of finger poses for a specific user can be obtained. Then, a general temporal model can be trained based on the specific set of touch screen images and the specific set of finger poses to obtain a temporal finger pose estimation model. This can reduce the impact of individual differences in fingers on finger pose prediction and improve the accuracy of finger pose prediction.

[0098] In one possible implementation, the error between the finger sequence pose and the finger pose training set is calculated, including:

[0099] The error between the finger sequence pose and the finger pose training set is calculated using a preset loss function. The formula for the preset loss function is as follows:

[0100]

[0101] Where, α k This represents the training value of the deflection angle corresponding to the k-th frame image. β represents the predicted deflection angle corresponding to the k-th frame image. k This represents the training value of the pitch angle corresponding to the k-th frame image. L represents the predicted pitch angle value corresponding to the k-th frame image. MSE Let N represent the mean squared error loss function, and let N represent the sequence length of the touchscreen image training set.

[0102] In this embodiment, the error between the finger sequence pose and the finger pose training set can be calculated using a preset loss function. The formula for the preset loss function is as follows:

[0103]

[0104] Where, α k This represents the training value of the deflection angle corresponding to the k-th frame image. β represents the predicted deflection angle corresponding to the k-th frame image. k This represents the training value of the pitch angle corresponding to the k-th frame image. L represents the predicted pitch angle value corresponding to the k-th frame image. MsE Let N represent the mean squared error loss function, and let N represent the sequence length of the touchscreen image training set.

[0105] Thus, the error between the finger sequence pose and the finger pose training set can be calculated according to the formula of the above-mentioned preset loss function, thereby obtaining the trained general temporal model. Predicting finger pose through the general temporal model can improve the accuracy of finger pose prediction.

[0106] To achieve the above embodiments, such as Figure 7As shown, this embodiment also provides a finger posture estimation device 700 based on a touch screen. The device 700 includes: a first acquisition module 710 and an output module 720.

[0107] The first acquisition module 710 is used to acquire the touchscreen image of the user's finger.

[0108] The output model 720 is used to input the touch screen image into the pre-trained temporal finger pose estimation model and output the finger pose corresponding to the touch screen image. The temporal finger pose estimation model includes an image feature extraction network, a temporal feature fusion network, and a finger pose prediction network.

[0109] In one possible implementation, the touchscreen-based finger pose estimation device 700 further includes:

[0110] The second acquisition module is used to acquire a touch screen image training set of at least one training user's finger and a finger posture training set corresponding to the touch screen image training set;

[0111] The first training module is used to train a general temporal model to be trained based on the touch screen image training set and the finger posture training set, and to obtain the trained general temporal model.

[0112] The third acquisition module is used to acquire a specific touch screen image training set for a specific user's finger and a specific finger posture training set corresponding to the specific touch screen image training set;

[0113] The second training module is used to fine-tune the general temporal model based on a specific touchscreen image training set and a specific finger pose training set, and to obtain a pre-trained temporal finger pose estimation model.

[0114] In one possible implementation, the first training module includes:

[0115] The first output unit is used to input the touch screen image training set into the single frame image feature extraction module and output the single frame image features.

[0116] The second output unit is used to input single-frame image features into the temporal feature fusion module and output image sequence features;

[0117] The third output unit is used to input image sequence features into the finger pose prediction module and output the finger sequence pose.

[0118] The computational unit is used to calculate the error between the finger sequence pose and the finger pose training set;

[0119] The training unit is used to train the time series model to be trained using the gradient descent algorithm based on the error, and obtain the trained general time series model.

[0120] In one possible implementation, the third acquisition module includes:

[0121] The acquisition unit is used to acquire a specific touch screen image training set and a specific finger posture training set by instructing a specific user to lift their finger from a flat position to an upright position in the direction of the arrow displayed on the interface.

[0122] In one possible implementation, the computing unit includes:

[0123] The computational subunit is used to calculate the error between the finger sequence pose and the finger pose training set using a preset loss function. The formula for the preset loss function is as follows:

[0124]

[0125] Where, α k This represents the training value of the deflection angle corresponding to the k-th frame image. β represents the predicted deflection angle corresponding to the k-th frame image. k This represents the training value of the pitch angle corresponding to the k-th frame image. L represents the predicted pitch angle value corresponding to the k-th frame image. MSE Let N represent the mean squared error loss function, and let N represent the sequence length of the touchscreen image training set.

[0126] According to the finger pose estimation device based on a touchscreen according to embodiments of this application, a first acquisition module acquires a touchscreen image of a user's finger; then, an output module inputs the touchscreen image into a pre-trained temporal finger pose estimation model and outputs the finger pose corresponding to the touchscreen image. The temporal finger pose estimation model includes an image feature extraction network, a temporal feature fusion network, and a finger pose prediction network. This application trains a temporal finger pose estimation model based on continuous touchscreen images of the finger to predict finger poses, which can improve the accuracy of finger pose prediction and enrich human-computer interaction input modes.

[0127] It should be noted that the foregoing explanation of the embodiment of the touch screen-based finger pose estimation method also applies to the touch screen-based finger pose estimation device of this embodiment, and will not be repeated here.

[0128] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0129] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0130] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A finger pose estimation method based on a touchscreen, characterized in that, include: To acquire the touchscreen image of the user's finger, the device's touchscreen driver is modified so that when the user operates the device by touching the touchscreen with their finger, the device outputs the original capacitance matrix to acquire the touchscreen image of the user's finger. The touch screen image is input into a pre-trained temporal finger pose estimation model, which outputs the finger pose corresponding to the touch screen image. The temporal finger pose estimation model includes an image feature extraction network, a temporal feature fusion network, and a finger pose prediction network. The temporal feature fusion network uses a self-attention mechanism to perform temporal modeling on the single-frame image features corresponding to multiple consecutive touch screen images, and outputs image sequence features that fuse temporal information. Before inputting the touchscreen image into a pre-trained temporal finger pose estimation model and outputting the finger pose corresponding to the touchscreen image, the following steps are included: Obtain a training set of touchscreen images of at least one training user's fingers and a training set of finger postures corresponding to the training set of touchscreen images; Train a general temporal model to be trained based on the touch screen image training set and the finger posture training set, and obtain the trained general temporal model; A training set of specific touchscreen images of a specific user's finger and a training set of specific finger postures corresponding to the specific touchscreen image training set are obtained. This is achieved by instructing the specific user to lift their finger from a flat position to an upright position according to the arrow direction displayed on the interface. Furthermore, application software specifically designed to record touchscreen images and corresponding finger postures is developed. The software interface displays registered deflection angle direction arrows. Following the prompts, the user uniformly lifts their finger from a flat position to an upright position. The arrow direction during this process is used as the pseudo-true value of the deflection angle, and N values ​​obtained by uniform interpolation within a range are used as the pseudo-true values ​​of the pitch angle. The general temporal model is fine-tuned based on the specific touchscreen image training set and the specific finger pose training set to obtain the pre-trained temporal finger pose estimation model.

2. The finger pose estimation method based on a touchscreen according to claim 1, characterized in that, The step of training a general temporal model based on the touchscreen image training set and the finger posture training set, and obtaining the trained general temporal model, includes: The touchscreen image training set is input into the single-frame image feature extraction module, and the single-frame image features are output. The single-frame image features are input into the temporal feature fusion module, which outputs image sequence features. The image sequence features are input into the finger pose prediction module, which outputs the finger pose sequence. Calculate the error between the finger sequence pose and the finger pose training set; Based on the error, the time series model to be trained is trained using the gradient descent algorithm to obtain the trained general time series model.

3. The finger pose estimation method based on a touchscreen according to claim 2, characterized in that, The calculation of the error between the finger sequence pose and the finger pose training set includes: The error between the finger sequence pose and the finger pose training set is calculated using a preset loss function, wherein the formula for the preset loss function is as follows: in, This represents the training value of the deflection angle corresponding to the k-th frame image. This represents the predicted deflection angle value corresponding to the k-th frame image. This represents the training value of the pitch angle corresponding to the k-th frame image. This represents the predicted pitch angle value corresponding to the k-th frame image. This represents the mean squared error loss function. N This indicates the sequence length of the touchscreen image training set.

4. A finger pose estimation device based on a touch screen, characterized in that, include: The first acquisition module is used to acquire the touch screen image of the user's finger. The device's touch screen driver is modified so that when the user operates the device by touching the touch screen with their finger, the device outputs the original capacitance matrix to acquire the touch screen image of the user's finger. The output module is used to input the touch screen image into a pre-trained temporal finger pose estimation model and output the finger pose corresponding to the touch screen image. The temporal finger pose estimation model includes an image feature extraction network, a temporal feature fusion network and a finger pose prediction network. The temporal feature fusion network uses a self-attention mechanism to perform temporal modeling on the single-frame image features corresponding to multiple consecutive touch screen images and outputs image sequence features with fused temporal information. The device further includes: The second acquisition module is used to acquire a touch screen image training set of at least one training user's finger and a finger posture training set corresponding to the touch screen image training set; The first training module is used to train a general temporal model to be trained based on the touch screen image training set and the finger posture training set, and to obtain the trained general temporal model. The third acquisition module is used to acquire a specific touchscreen image training set of a specific user's finger and a specific finger posture training set corresponding to the specific touchscreen image training set. Specifically, the specific touchscreen image training set and the specific finger posture training set are acquired by instructing the specific user to lift their finger from a flat position to an upright position according to the arrow direction displayed on the interface. Furthermore, application software specifically designed to record touchscreen images and corresponding finger postures is developed. The software interface displays a registered deflection angle direction arrow. Following the prompts, the user evenly lifts their finger from a flat position to an upright position. The arrow direction during this process is used as the pseudo-true value of the deflection angle, and N values ​​obtained by uniform interpolation within a range are used as the pseudo-true value of the pitch angle. The second training module is used to fine-tune the general temporal model based on the specific touch screen image training set and the specific finger posture training set, and obtain the pre-trained temporal finger posture estimation model.

5. The finger pose estimation device based on a touchscreen according to claim 4, characterized in that, The first training module includes: The first output unit is used to input the touch screen image training set into the single frame image feature extraction module and output the single frame image features. The second output unit is used to input the single-frame image features into the temporal feature fusion module and output image sequence features; The third output unit is used to input the image sequence features into the finger pose prediction module and output the finger sequence pose. A computing unit is used to calculate the error between the finger sequence pose and the finger pose training set; The training unit is used to train the time-series model to be trained using a gradient descent algorithm based on the error, and obtain the trained general time-series model.

6. The finger pose estimation device based on a touchscreen according to claim 5, characterized in that, The computing unit includes: The computational subunit is used to calculate the error between the finger sequence pose and the finger pose training set using a preset loss function, wherein the formula for the preset loss function is as follows: in, This represents the training value of the deflection angle corresponding to the k-th frame image. This represents the predicted deflection angle value corresponding to the k-th frame image. This represents the training value of the pitch angle corresponding to the k-th frame image. This represents the predicted pitch angle value corresponding to the k-th frame image. This represents the mean squared error loss function. N This indicates the sequence length of the touchscreen image training set.

Images