Finger identification method, error correction method and system based on capacitive screen

Abstract

The invention discloses a capacitive screen-based finger identification method and system, and an error correction method and system, and belongs to the technical field of human-computer interaction. The method comprises the following steps: acquiring original touch data of a capacitive screen, extracting and tracking a finger contact area, and generating a single-finger capacitance image sequence; and extracting space and time sequence features of the image by using a deep neural network of an encoder-decoder architecture to complete finger category identification. And for a misrecognition result, the user can activate an off-screen or non-off-screen error correction mode, finger recognition is carried out again, a wrong result is corrected, and misoperation is recovered. The problem of insufficient accuracy of traditional finger recognition is solved, flexible correction of error results is supported, withdrawing and repeated input operations are reduced, the recognition precision and user experience are improved, the model can be iteratively optimized through the recognition result, and the subsequent recognition effect is further improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of human-computer interaction, and particularly relates to a finger recognition method, an error correction method and a system based on a capacitive screen. Background Art

[0002] Touch input has already become a common interaction method for electronic devices in daily life. The touch screen integrates the functions of input and output at the same time, thus providing users with a natural and dynamic interaction method. However, at the same time, the touch input itself has limited touch input space because the touch reporting point data is only limited to two-dimensional coordinates, ignoring the rich information of the hand and fingers in the interaction.

[0003] In the technical field of human-computer interaction, many related works have enriched the touch input dimension by obtaining the source of input (such as distinguishing fingers, palms, nails, knuckles, ears, etc.) and the state of input (such as the about-to-touch state, finger shape, touch pressure, finger posture, finger angle, etc.), so as to support a wider range of interactions. Among them, finger recognition technology supports distinguishing the inputs of different fingers on the capacitive screen, so as to support assigning different functions to fingers and realizing more interaction methods. Currently, the main four types of sensors used in finger recognition technology are: visual sensors, wearable sensors, acoustic sensors, and capacitive sensors. Visual sensors need to rely on external cameras or under-screen cameras, which are inconvenient in terms of hardware integration and use. Wearable sensors require users to wear devices such as bracelets and rings, which are also inconvenient in use. Acoustic sensors can directly utilize the built-in speakers and microphones of the device, with good hardware integration, but there are problems such as a limited set of recognizable gestures and uncertain robustness under different environmental noise conditions. Using the original capacitance data of the capacitive screen does not require additional devices and is not easily affected by the external environment, and has good application prospects in the current situation where capacitive sensor-based touch screens are widely used.

[0004] Capacitive sensors usually sample input signals in the form of row and column matrices, so the raw data of capacitance can be viewed as a grayscale image, usually referred to as a "capacitive image". Limited by the hardware configuration of the capacitive screen, the resolution of the "capacitive image" collected by the capacitive sensor is low, with a pixel size of about 5mm*5mm. This resolution is sufficient to accurately obtain two-dimensional touch input or stylus input coordinate points, but it is not enough to complete tasks such as finger recognition that require more refined features. In order to overcome the challenges of low resolution to finger recognition, there are two main solutions in existing work. One is to require users to deliberately produce exaggerated input gestures to enhance the recognition effect, and the other is to use a smaller set of recognizable fingers to enhance the recognition effect in a limited set. The recognition methods of existing work all rely on a single "capacitive image" to complete the recognition through machine learning or deep learning models, and lack the use of information that may be implicit in the image sequence generated during the action process.

[0005] Although the accuracy of the finger recognition models in existing work has basically reached a usable level, there is still an inevitable problem of reduced recognition effect in actual use, which will lead to errors in the results and cause the operation to be inconsistent with the intention. The commonly used method is to set a withdrawal method for reversible operations, and correct the errors caused by the recognition through the process of withdrawal and re-entering. However, since re-entry still needs to rely on the recognition results of the finger recognition model, if the recognition is wrong again, the user will repeat this process many times, which will bring frustration to the user and affect the user experience. Therefore, it is very necessary to provide a convenient and accurate correction method for the errors caused by the misrecognition results, which is beneficial to improving the interactive experience of finger recognition interaction. Summary of the invention

[0006] The purpose of the present invention is to provide a capacitive screen-based finger recognition method, error correction method and system, which accurately classifies finger categories through a deep model and improves recognition accuracy, aiming to solve the limitations of low resolution of capacitive screens on finger recognition, as well as the problem that existing methods do not fully utilize image sequence timing information and have low error correction efficiency.

[0007] The technical solution adopted by the present invention to achieve its purpose is as follows:

[0008] A capacitive touch screen based finger recognition method comprises the following steps:

[0009] Collect capacitance data generated by fingers touching the capacitive screen and generate capacitance data sequences for each finger;

[0010] Mapping the capacitance data sequence to generate a corresponding capacitance image sequence;

[0011] Input the capacitive image sequence frame by frame into the encoder, extract the spatial features of each frame image, and generate the hidden layer representation;

[0012] Input the hidden layer representation of the capacitance image sequence into the decoder, extract the temporal features of the capacitance image sequence through temporal modeling, and generate a hidden layer feature vector;

[0013] Input the hidden layer feature vector into the classifier, calculate the probability distribution of the finger categories, and output the finger category with the highest probability as the finger recognition result;

[0014] Use the capacitance images with finger category labels as the training data set, iteratively train and update the parameters of the encoder, decoder, and classifier to minimize the cross-entropy loss function.

[0015] Further, the steps of generating the capacitance data sequence include:

[0016] For the collected capacitance data, use the connected component detection algorithm to extract the contact area blocks in each frame of capacitance data;

[0017] Based on the spatio-temporal correlation of the contact area blocks between multiple frames, perform finger tracking, and classify the contact area blocks with an intersection greater than the threshold in adjacent frames as the same finger to generate the capacitance data sequence of each finger.

[0018] Further, the steps of generating the capacitance image sequence include:

[0019] Linearly map the capacitance data in the capacitance data sequence to gray values to generate the gray capacitance image of each frame;

[0020] Preprocess the gray capacitance image, and interpolate or sample the image sequence to obtain a fixed-length capacitance image sequence.

[0021] Further, the preprocessing includes image scaling and normalization, the interpolation uses linear interpolation or bilinear interpolation, and the sampling uses average sampling.

[0022] Further, the encoder uses, but is not limited to, a convolutional neural network.

[0023] Further, the decoder uses, but is not limited to, a recurrent neural network.

[0024] Further, the classifier uses, but is not limited to, a fully connected layer.

[0025] A finger recognition error correction method based on a capacitive screen, which performs finger recognition based on the above finger recognition method, includes the following steps:

[0026] Based on the capacitance data generated by the finger touching the capacitive screen, divide the contact area blocks of the finger, track the contact area blocks of consecutive frames, and identify their states;

[0027] According to the state of the contact area block, determine whether the finger recognition trigger condition is satisfied. If it is satisfied, perform finger recognition;

[0028] During the finger recognition process, based on the sequence information of the contact area blocks of all fingers, it is determined whether the triggering condition for entering the error correction mode is satisfied. If it is satisfied, it is determined whether to enter the off-screen error correction mode or the on-screen error correction mode, and finger recognition error correction is performed;

[0029] If the triggering condition for entering the error correction mode is not satisfied, it is determined one by one whether there is a sequence of contact area blocks that satisfies the finger recognition triggering condition. If there is, finger recognition is performed and a finger touch event is triggered. Otherwise, the finger touch event is directly triggered;

[0030] After triggering the finger touch event, the finger touch event parameters are recorded and the processing result of the current frame is saved for use in subsequent frame processing.

[0031] Furthermore, the states of the contact area blocks include:

[0032] Just touched the screen: The current frame is the first frame, or there is no contact area block in the previous frame or multiple previous frames that matches the contact area block of the current frame;

[0033] Continuing to maintain screen contact: There is a contact area block in the previous frame or multiple previous frames that matches the contact area block of the current frame; or

[0034] Has left the screen contact: The contact area blocks in the previous frame or multiple previous frames do not match any of the contact area blocks of the current frame.

[0035] Furthermore, the finger recognition triggering conditions include:

[0036] The number of data frames or duration of the contact area blocks of a certain finger reaches a preset threshold; or

[0037] The finger has left the screen and finger recognition has not been completed.

[0038] Furthermore, the triggering conditions for entering the error correction mode include:

[0039] Currently, there are 5 or 10 finger contact area blocks touching the screen simultaneously; or

[0040] The finger touch intensity is greater than the preset threshold or the finger touch duration is greater than the preset threshold.

[0041] Furthermore, the steps for determining whether to enter the off-screen error correction mode or the on-screen error correction mode include:

[0042] If the screen was not in the finger input state before entering the error correction mode and there is already a finger recognition result, it is determined as the off-screen error correction mode:

[0043] If the screen is in the finger input state and there is already a finger recognition result when entering the error correction mode, it is determined as the on-screen error correction mode.

[0044] Further, the steps for finger recognition error correction include:

[0045] 1) If it is currently in the two - hand ten - finger error correction mode, divide the two hands and the contact area blocks of 5 fingers on each hand according to the finger positions;

[0046] 2) For each hand, distinguish the thumb, little finger and other fingers according to the contact area block data of the five fingers;

[0047] 3) Based on the distinction results of the thumb and little finger obtained in step 2), complete the polar - angle sorting of the other fingers with the thumb as the center. If the little finger has the smallest value in the sorting result, it belongs to the left hand; if the little finger has the largest value in the sorting result, it belongs to the right hand;

[0048] 4) Combine the distinction results of the thumb and little finger obtained in step 2) and the left - hand / right - hand judgment results obtained in step 3), determine the categories of the other fingers according to the polar - angle sorting, and finally obtain the specific categories of each finger on the left and right hands; the specific categories are described according to the finger order, including: the finger order of the left hand is little finger, ring finger, middle finger, index finger, thumb, and the finger order of the right hand is thumb, index finger, middle finger, ring finger, little finger.

[0049] Further, the method for dividing according to finger positions is: calculate the minimum - bounding rectangle of the contact area blocks of 10 fingers, sort the finger positions in any direction of the long side of the rectangle, and divide the first 5 fingers and the last 5 fingers into two hands respectively.

[0050] Further, the steps for distinguishing the thumb, little finger and other fingers according to the contact area block data of the five fingers include:

[0051] Calculate the maximum included angle formed by the vertex of each finger and any other two vertices, and select the two fingers with the smallest included angle after sorting as the candidate fingers for the thumb and little finger;

[0052] Assign weights to the relationship between the candidate fingers and the other three fingers and sort them, select the one with the larger weighted sum as the thumb and the one with the smaller weighted sum as the little finger.

[0053] Further, when assigning weights to the relationship between the candidate fingers and the other three fingers, the weight calculation methods include:

[0054] The maximum distance from the candidate finger to the other three fingers;

[0055] The sum of the distances from the candidate finger to the other three fingers;

[0056] The maximum included angle formed by the vector between the candidate fingers and the vectors from the candidate fingers to the other three fingers; or

[0057] The area of the minimum - bounding box of the contact area block of the candidate finger.

[0058] Further, the finger touch event parameters include: timestamp, finger contact position, touch area block capacitance data, whether in the error correction mode, whether the finger has been recognized, and whether finger recognition is enabled.

[0059] Further, the processing result of the current frame includes: touch area block sequence, touch area block capacitance data, and finger recognition result.

[0060] A finger recognition and error correction system based on a capacitive screen, comprising the following modules:

[0061] Capacitive sensor module: used to collect capacitance data generated by a finger touching the capacitive screen in real time, generate an original capacitance data sequence; divide the touch area block of the finger based on the collected capacitance data, and output the area block state of each frame.

[0062] Display module: used to display the finger recognition result, including finger category, touch event, and related parameters; when entering the error correction mode, prompt the user for the error correction status, including off-screen error correction mode or on-screen error correction mode; used to display the error correction result and other interface elements to help the user confirm or adjust the finger category and touch event recognized by the system.

[0063] Calculation module: used for finger recognition calculation, error correction mode calculation, and dynamic event judgment; among them, the finger recognition calculation includes: mapping the capacitance data sequence into a capacitance image sequence, inputting each frame into the encoder of the neural network model frame by frame, extracting the spatial features of each frame of capacitance image, and generating a hidden layer representation; inputting the hidden layer representation into the decoder, extracting the temporal features of the capacitance image sequence through temporal modeling, and generating a hidden layer feature vector; inputting the hidden layer feature vector into the classifier, calculating the probability distribution of the finger category, and outputting the finger category with the highest probability as the finger recognition result; the error correction mode calculation includes: judging whether the finger recognition trigger condition is satisfied according to the state of the touch area block, triggering finger recognition; judging whether the trigger condition for entering the error correction mode is satisfied, further distinguishing between off-screen error correction mode or on-screen error correction mode, and recalculating the finger category and touch event in the error correction mode; the dynamic event judgment includes: if the finger touch event does not require error correction, judging whether there is a touch area block sequence that satisfies the finger recognition trigger condition, and directly triggering the corresponding event.

[0064] Storage module: used to save the execution program, including the operation logic of finger recognition and error correction; store the neural network model for finger recognition and its training parameters; save the finger touch event parameters, recognition results, error correction results, and finger touch area block information.

[0065] The technical effects achieved by the present invention are as follows:

[0066] 1. Efficient Finger Recognition and Classification Model: The present invention designs an artificial neural network model based on the encoder-decoder architecture for processing the finger motion capacitance image sequences collected on a capacitive screen. By utilizing the spatial features and sequence features of the input capacitance images, the model can accurately identify the finger categories of five fingers of one hand or ten fingers of both hands. Through the classification and attribute analysis of the finger contact area blocks, the model has high accuracy and versatility, and can adapt to the changes in different gestures, touch areas, and capacitance image qualities.

[0067] 2. Intelligent Misrecognition Correction Mechanism: The present invention proposes an interactive finger recognition correction method that allows users to trigger the correction mode through simple operations, automatically obtain the previously input capacitance data, and re-perform finger recognition. The correction method supports the input modes of five fingers of one hand or ten fingers of both hands, without being limited to the number of fingers or the recognition range. Only one correction operation is required to correct the misrecognition results, avoiding the cumbersome operations of users having to withdraw and re-enter multiple times due to incorrect classification. The generated correction results can be consistent with the actual input of users, and further improve the model performance through feedback data.

[0068] 3. Optimization of the Model's Continuous Learning Ability: During the finger recognition and correction process, the present invention can collect the capacitance data and their labels in actual use, and use these data for the incremental training of the model to improve the robustness and accuracy of the recognition model. The data-driven optimization mechanism enables the model to continuously adapt and improve in different application scenarios, thereby enhancing the long-term effectiveness of the model.

[0069] 4. User-Friendly Interaction Experience: The present invention integrates the finger recognition and correction processes into a single device of the capacitive screen, without the need for additional hardware or complex operations, providing a highly integrated interaction experience. It supports finger recognition and correction for short actions (such as clicks and swipes) and long actions (such as drags and presses), meeting the diverse user scenario requirements and improving the usability and interaction efficiency.

[0070] 5. Wide Application Scenarios and Hardware Adaptability: Due to the wide application of capacitive screens in fields such as mobile devices, smart homes, and industrial control, the present invention has good practical promotion value. Based on the high integration and data acquisition convenience of capacitive screens, it can significantly improve the human-computer interaction capabilities of related devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 is the structural diagram of the finger recognition model based on the encoder-decoder.

[0072] Figure 2 is an example diagram of the capacitance image sequence.

[0073] Figure 3 is the flow schematic diagram of the finger recognition correction method based on the capacitive screen.

[0074] Figure 4 This is the schematic diagram of the two modes of finger recognition error correction without leaving the screen. DETAILED DESCRIPTION

[0075] In order to make the various technical features and advantages or technical effects in the above technical solutions of the present invention more obvious and easy to understand, they are described in detail below with reference to the accompanying drawings.

[0076] An embodiment of the present invention specifically discloses a finger recognition method based on a capacitive screen. The method converts the capacitance data generated by a finger touching the capacitive screen into a capacitive image sequence data, and then classifies it through a finger recognition model based on an encoder-decoder, thereby identifying the finger. The specific description is as follows.

[0077] Since the capacitive sensor on the capacitive screen is usually fixed in length and width W F ×h F are arranged in a matrix form with a fixed interval length Δt F Refresh acquisition data (Δt F Related to the capacitive sensor), so the capacitance data obtained is also the corresponding length and width matrix F t , t represents the tth frame data. Matrix F t Elements in It is a non-negative number, representing the data of a certain capacitance point of the capacitive sensor. When a finger contact signal is obtained otherwise In the time length T, the capacitance data obtained is F = {F 1 , F 2 , ..., F M},in Indicates the number of capacitance frames obtained in the duration T, F M Represents the capacitance data of the Mth frame.

[0078] 1) Preprocessing of capacitance data

[0079] First, a connected domain detection algorithm, such as a blob detection algorithm, is used to extract connected domains in each frame of capacitance data, each of which represents a contact area block generated by a finger on the frame image.

[0080] Secondly, the contact area blocks between multiple frames are tracked Tr. If the intersection area of ​​the contact area block of the current frame and the contact area block of the adjacent frame is greater than a preset threshold, the contact area blocks of the current frame and the adjacent frame are classified as being generated by the same finger. Based on this, the capacitance data sequence generated by all finger touches can be calculated. Among them, finger 1 , finger 2,... finger k represents k fingers touching the screen. The generation of this capacitance data sequence can be expressed in the following form:

[0081] S = Tr(B(F 1 ), B(F 2 ),..., B(F M ))

[0082] where B(F i ) represents all the contact area blocks generated by all fingers in a certain frame.

[0083] The capacitance data sequence of each finger is also composed of the capacitance data BF of the contact area blocks of this finger in each frame, that is where and represent the start and end frames when finger finger i touches the screen respectively.

[0084] 2) Conversion of capacitance data into capacitance image sequence

[0085] Converting the capacitance data sequence of each finger into a capacitance image sequence includes the following steps:

[0086] For finger finger i , the capacitance data of each frame of the finger contact area block (st ≤ j ≤ ed, j is the contact frame), according to the value range of the original capacitance data element , is linearly mapped to the pixel gray value range of the gray image, and the value of each capacitance data is mapped to the p×q area of the capacitance image, that is, all areas in this image have the same gray value. Thus, the length and width of the gray capacitance image are W FF ×p and the length is h FF ×q respectively, and the corresponding gray capacitance image

[0087] Obviously, the size of each frame of finger capacitance image is not necessarily the same. Therefore, necessary preprocessing R such as image scaling and normalization is performed on each frame of the image to increase the image data. Then, interpolation (such as linear interpolation, bilinear interpolation) or sampling P (such as average sampling) is performed on the complete image sequence to obtain a sequence of length N, which is the same length as the input capacitance image sequence. The processing of the capacitance data sequence of each finger to the capacitance image sequence can be expressed as:

[0088]

[0089] where represents finger fingeri The generated capacitance image sequence.

[0090] 3) Perform finger recognition based on the encoder-decoder architecture

[0091] In the capacitance image sequence data, the spatial information of a single-frame image and the sequence information between multiple-frame images contribute to identifying specific finger categories by leveraging the characteristics exhibited during the finger touch process. Therefore, a finger recognition neural network model based on an encoder-decoder can be used to achieve the identification of finger categories. The architecture of this finger recognition model is as Figure 1 shown, including an encoder (Encoder), a decoder (Decoder), and a classifier (Classification). The specific processing flow is as follows.

[0092] (1) The encoder performs encoding

[0093] The input to the finger recognition model is a capacitance image sequence X = {x 1 , x 2 ,..., x N} that has been processed to be of equal length N and the same image size, where x t (t = 1, 2,..., N) represents the capacitance image of the t-th frame, and the size of each frame image is W X × h X , as shown in the example of Figure 2 , where x 1 - x 30 represent 30-frame capacitance images. Each frame image x t is input into the same encoder, which captures the spatial features of the finger input in a single-frame image and represents the original input image as a hidden layer representation e t of the spatial features, and the dimension of this representation is d e . This encoder can be selected from but not limited to convolutional neural networks. For example, AlexNet, VGGNet, ResNet, etc. can be used. Figure 1 The encoder in takes the network structure of AlexNet as an example. The encoder encoding can be expressed as:

[0094] e t = Encoder(x t )

[0095] (2) The decoder performs decoding

[0096] Input the hidden layer representations e t of the spatial features of all single-frame images into a unified decoder, and the decoder learns the sequence features in the finger input. The input format of the decoder is d e×N, where N is the step size of the decoder in the time dimension, which is consistent with the fixed length of the input capacitance image sequence. The decoder can be selected from, but not limited to, recurrent neural networks. For example, a long short-term memory network (LSTM) can be used. Figure 1 The decoder in

[0097] takes the network structure of LSTM as an example. The decoding of the decoder can be expressed as: t , h t )(c t-1 , h t-1 , e t )

[0098] Among them, the memory cell c t and the hidden layer unit h t are the function values based on c t-1 , h t-1 generated by the capacitance image of the (t - 1)-th frame and e t of the input sample. Finally, the hidden layer feature vector H = [h 1 , h 2 ,..., h N is obtained.

[0099] (3) The classifier performs finger recognition

[0100] Input the hidden layer feature vector H into a classifier, and the output is the probability value distribution of different finger categories Among them, m represents the number of finger categories to be classified, and k i (i = 1, 1,..., m) represents each finger category to be classified. The structure of the classifier can be selected from one or more fully connected layers. Figure 1 In

[0101] the classifier takes the structure of a single-layer fully connected network as an example. The probability of the finger category to be classified is the function value of the hidden layer feature vector, which is expressed as:

[0102] After calculating the generalization value distribution, the finger category with the highest probability is used as the classification result, which can be expressed as That is, the input capacitance image sequence is recognized and classified as finger k F .

[0103] (4) Finger recognition model training

[0104] Before the finger recognition model based on the encoder-decoder is formally used, it needs to be trained first. The training dataset used is expressed as D = [X in , Y label , X inis capacitance image data, Y label is finger category label data. The overall training objective of this model is measured by the cross-entropy loss function L(D; θ e , θ d , θ c ), where θ e , θ d and θ c are the parameters of the encoder, decoder, and classifier respectively. The model is iteratively trained on the training dataset to update the parameters θ e , θ d , θ c to minimize L. For each training sample, the loss function L(D; θ e , θ d , θ c ) can be expressed as:

[0105]

[0106] where x ∈ X in is a capacitance image sequence sample, y ∈ Y label is the corresponding finger category label; x[y] indicates whether the x sample belongs to the y class, 1 for belonging and 0 for not belonging; the base of the log is e.

[0107] Based on the above finger recognition method, an embodiment of the present invention also specifically discloses a finger recognition error correction method based on a capacitive screen, which is used to conveniently correct the error results of finger recognition. The entire process is shown in Figure 3 . For ease of description, two roles, user and system, are introduced. The system refers to the software and hardware device that runs this finger recognition error correction method, and the specific description is as follows:

[0108] 1. User input step:

[0109] The user makes an input on the capacitive screen, generating capacitance data of the finger touching the capacitive screen.

[0110] 2. Finger recognition step:

[0111] When the user completes the input action (such as leaving the screen or reaching the input duration threshold), the system performs a finger recognition operation, and recognizes the user's finger action by tracking the contact area block of consecutive frames, including the following three parallel states:

[0112] a. Just touching the screen: The current frame is the first frame, or the contact area block of the current frame does not match the contact area block of the previous frame or multiple previous frames, and starts to record the position and timestamp information of the contact area block of the current frame.

[0113] b. Continuing to maintain screen contact: The contact area blocks of the current frame match those of the previous frame or multiple previous frames, and the position and timestamp information of the contact area blocks of the current frame continue to be recorded.

[0114] c. Having left the screen contact: Any contact area block of the current frame does not match the contact area blocks of the previous frame or multiple previous frames, and recording stops.

[0115] For finger actions of continuing to maintain screen contact or having left the screen contact, it is determined whether the finger recognition trigger condition is satisfied. If the finger recognition trigger condition is satisfied, the finger category and action are recognized, and relevant event parameters are generated, including information such as finger recognition results, finger actions, and timestamps. If the user has ended the input, the system triggers a finger recognition event based on the recognition result and performs corresponding operations.

[0116] 3. Finger recognition error correction steps:

[0117] If the user discovers an identification error, the error correction mode can be triggered in the following ways:

[0118] a. Place all five fingers of one hand or all ten fingers of both hands spread out on the screen.

[0119] b. If the screen has been left previously, the error correction can be triggered by lifting the finger and then placing it back on the screen.

[0120] c. If the finger remains in contact, directly enter the error correction mode and wait for the result.

[0121] The system determines whether the relevant information of all contact area block sequences satisfies the trigger condition for entering the error correction mode. If it is satisfied, all finger contact area block information is input into the error correction method to re-recognize the finger set.

[0122] When the finger touches the screen, it can be divided into two modes: off-screen and on-screen. Figure 4 It is the schematic diagram of finger recognition error correction in these two off-screen and on-screen modes. Among them, the off-screen mode is mainly common in short actions (such as clicking, swiping, etc.), and the on-screen mode is mainly common in long actions (such as pressing, dragging, etc.). When the user discovers an identification error after lifting the hand during a long action, the off-screen mode can also be used to complete the error correction. The processes of finger recognition error correction for these two modes will be described in detail.

[0123] (1) Off-screen mode: The user touches and inputs on the screen at time T 0 and ends the input before the input duration threshold (T 1 -T 0 ); after the finger leaves the screen at time T 1 , the system performs finger recognition based on the input capacitance data (the recognition process occurs at T 1 -T 2(Time period); After completing finger recognition and obtaining the result, the system will display a visual prompt on the screen interface and perform corresponding operations at time T according to the recognition result; After the user finds that the recognition result is inconsistent with the intention, manually activate the off-screen error correction mode at time T; The system starts the off-screen error correction mode and completes finger recognition in the error correction mode within the time period from T to T; After obtaining the corrected finger recognition result, prompt it to the user at time T; When the error correction result is consistent with the intention, end the input, and the system will automatically re-execute the previous operation according to the error correction result at time T. 2 At time T, 3 After the user finds that the recognition result is inconsistent with the intention, manually activate the off-screen error correction mode at time T; The system starts the off-screen error correction mode and completes finger recognition in the error correction mode within the time period from T to T; After obtaining the corrected finger recognition result, prompt it to the user at time T; When the error correction result is consistent with the intention, end the input, and the system will automatically re-execute the previous operation according to the error correction result at time T. 3 -T 4 After obtaining the corrected finger recognition result, prompt it to the user at time T; When the error correction result is consistent with the intention, end the input, and the system will automatically re-execute the previous operation according to the error correction result at time T. 4 When the error correction result is consistent with the intention, end the input, and the system will automatically re-execute the previous operation according to the error correction result at time T. Figure 1 When the error correction result is consistent with the intention, end the input, and the system will automatically re-execute the previous operation according to the error correction result at time T. 5 When the error correction result is consistent with the intention, end the input, and the system will automatically re-execute the previous operation according to the error correction result at time T.

[0124] (2) Non-off-screen mode: The user touches and inputs on the screen at time T; After reaching the input duration threshold (from T to T), keep inputting. The system performs finger recognition at time T and shows the recognition result to the user at time T; When the recognition result is consistent with the intention, the user only needs to continue the input, and the system will perform the operation after the input ends. When they are inconsistent, the user manually activates the non-off-screen error correction mode at time T; The system starts the non-off-screen error correction mode and completes finger recognition in the error correction mode within the time period from T to T; After obtaining the corrected finger recognition result, prompt it to the user at time T; When the error correction result is consistent with the intention, the user can end the error correction mode and continue the input or directly end the input, and the system will automatically perform the operation according to the error correction result at time T. 0 At time T, 1 -T 0 After reaching the input duration threshold (from T to T), keep inputting. The system performs finger recognition at time T and shows the recognition result to the user at time T; When the recognition result is consistent with the intention, the user only needs to continue the input, and the system will perform the operation after the input ends. When they are inconsistent, the user manually activates the non-off-screen error correction mode at time T; The system starts the non-off-screen error correction mode and completes finger recognition in the error correction mode within the time period from T to T; After obtaining the corrected finger recognition result, prompt it to the user at time T; When the error correction result is consistent with the intention, the user can end the error correction mode and continue the input or directly end the input, and the system will automatically perform the operation according to the error correction result at time T. 1 -T 2 After reaching the input duration threshold (from T to T), keep inputting. The system performs finger recognition at time T and shows the recognition result to the user at time T; When the recognition result is consistent with the intention, the user only needs to continue the input, and the system will perform the operation after the input ends. When they are inconsistent, the user manually activates the non-off-screen error correction mode at time T; The system starts the non-off-screen error correction mode and completes finger recognition in the error correction mode within the time period from T to T; After obtaining the corrected finger recognition result, prompt it to the user at time T; When the error correction result is consistent with the intention, the user can end the error correction mode and continue the input or directly end the input, and the system will automatically perform the operation according to the error correction result at time T. 2 After reaching the input duration threshold (from T to T), keep inputting. The system performs finger recognition at time T and shows the recognition result to the user at time T; When the recognition result is consistent with the intention, the user only needs to continue the input, and the system will perform the operation after the input ends. When they are inconsistent, the user manually activates the non-off-screen error correction mode at time T; The system starts the non-off-screen error correction mode and completes finger recognition in the error correction mode within the time period from T to T; After obtaining the corrected finger recognition result, prompt it to the user at time T; When the error correction result is consistent with the intention, the user can end the error correction mode and continue the input or directly end the input, and the system will automatically perform the operation according to the error correction result at time T. Figure 1 When the recognition result is consistent with the intention, the user only needs to continue the input, and the system will perform the operation after the input ends. When they are inconsistent, the user manually activates the non-off-screen error correction mode at time T; The system starts the non-off-screen error correction mode and completes finger recognition in the error correction mode within the time period from T to T; After obtaining the corrected finger recognition result, prompt it to the user at time T; When the error correction result is consistent with the intention, the user can end the error correction mode and continue the input or directly end the input, and the system will automatically perform the operation according to the error correction result at time T. 3 When the recognition result is consistent with the intention, the user only needs to continue the input, and the system will perform the operation after the input ends. When they are inconsistent, the user manually activates the non-off-screen error correction mode at time T; The system starts the non-off-screen error correction mode and completes finger recognition in the error correction mode within the time period from T to T; After obtaining the corrected finger recognition result, prompt it to the user at time T; When the error correction result is consistent with the intention, the user can end the error correction mode and continue the input or directly end the input, and the system will automatically perform the operation according to the error correction result at time T. 3 -T 4 When the recognition result is consistent with the intention, the user only needs to continue the input, and the system will perform the operation after the input ends. When they are inconsistent, the user manually activates the non-off-screen error correction mode at time T; The system starts the non-off-screen error correction mode and completes finger recognition in the error correction mode within the time period from T to T; After obtaining the corrected finger recognition result, prompt it to the user at time T; When the error correction result is consistent with the intention, the user can end the error correction mode and continue the input or directly end the input, and the system will automatically perform the operation according to the error correction result at time T. 4 When the recognition result is consistent with the intention, the user only needs to continue the input, and the system will perform the operation after the input ends. When they are inconsistent, the user manually activates the non-off-screen error correction mode at time T; The system starts the non-off-screen error correction mode and completes finger recognition in the error correction mode within the time period from T to T; After obtaining the corrected finger recognition result, prompt it to the user at time T; When the error correction result is consistent with the intention, the user can end the error correction mode and continue the input or directly end the input, and the system will automatically perform the operation according to the error correction result at time T. Figure 1 When the error correction result is consistent with the intention, the user can end the error correction mode and continue the input or directly end the input, and the system will automatically perform the operation according to the error correction result at time T. 5 When the error correction result is consistent with the intention, the user can end the error correction mode and continue the input or directly end the input, and the system will automatically perform the operation according to the error correction result at time T.

[0125] Trigger the error correction mode by using both hands' ten fingers or one hand's five fingers, and perform finger recognition on the participating fingers, including the following steps:

[0126] (1) If the current is in the two-hands ten-fingers error correction mode, divide the 10 fingers into 5 fingers each according to the finger positions, which belong to two hands respectively. The finger positions can be calculated according to the center of gravity of the block capacitance data of the finger contact areas; The division method for two hands' ten fingers is to calculate the minimum bounding rectangle of the 10 finger positions, sort the finger positions in any direction along the long side of the rectangle, and consider the first 5 fingers and the last 5 fingers as two hands respectively.

[0127] (2) For each hand, distinguish the thumb, little finger, and other fingers from the data of the five fingers; specifically, the maximum angles formed by each finger vertex and any two other finger vertices can be sorted, and the 2 fingers with the smallest sorting results are taken as the thumb and little finger.

[0128] (3) Based on the relationship between the data of the two candidate fingers, the thumb and the little finger, and the other fingers, weights are assigned to the relationship between the thumb or little finger and the other three fingers, and they are sorted. The finger with the larger weighted sum is taken as the thumb, and the smaller one is taken as the little finger to distinguish the thumb and the little finger; there are the following four sorting methods:

[0129] a. The maximum distance from the thumb or little finger to the other three fingers;

[0130] b. The sum of the distances from the thumb or little finger to the other three fingers;

[0131] c. The maximum angle between the vector from the thumb or little finger to the little finger or thumb and the vector to any one of the other three fingers;

[0132] d. The minimum bounding box area of the touch area block capacitance data of the thumb or little finger.

[0133] (4) Based on the distinction results of the thumb and little finger, perform polar angle sorting on the other fingers with the thumb as the center. If the little finger has the smallest polar angle sorting result among them, then these five fingers belong to the left hand; if the little finger has the largest polar angle sorting result, then these five fingers belong to the right hand.

[0134] (5) Combining the left and right hand judgment results and the thumb and little finger distinction results, determine the specific finger categories for each finger according to the polar angle sorting results. The assignment order after sorting for the left hand is little finger, ring finger, middle finger, and index finger, and the assignment order after sorting for the right hand is index finger, middle finger, ring finger, and little finger.

[0135] 4. Re-execute the operation steps:

[0136] After obtaining the error correction result, the user can directly raise the hand to end the input, or raise the non-input finger to continue the input.

[0137] For short actions (such as clicking and swiping), error correction is completed in an off-screen manner; for long actions (such as pressing and dragging), error correction can also be performed in an off-screen manner after raising the hand.

[0138] 5. Trigger the finger recognition event:

[0139] After finger recognition or error correction is completed, trigger the finger touch event, and provide the relevant information of all fingers as event parameters to the subsequent processing logic of the system.

[0140] 6. Data saving and subsequent processing:

[0141] End the processing of the current frame, save the relevant contact area block data and recognition results for finger tracking and processing in subsequent frames.

[0142] Although the present invention has been disclosed above in embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent replacements made by those of ordinary skill in the art to the technical solutions of the present invention shall be covered within the protection scope of the present invention. The protection scope of the present invention shall be subject to that defined by the claims.

Claims

1. A finger recognition method based on a capacitive screen, characterized in that: The following steps are involved: Collect capacitance data generated by fingers touching the capacitive screen and generate capacitance data sequences for each finger; Mapping the capacitance data sequence to generate a corresponding capacitance image sequence; Input the capacitive image sequence frame by frame into the encoder, extract the spatial features of each frame image, and generate the hidden layer representation; The hidden layer representation of the capacitive image sequence is input into the decoder, and the temporal features of the capacitive image sequence are extracted through temporal modeling to generate a hidden layer feature vector; Input the hidden layer feature vector into the classifier, calculate the probability distribution of the finger category, and output the finger category with the highest probability as the finger recognition result; Using capacitive images with finger category labels as training datasets, the parameters of the encoder, decoder, and classifier are iteratively trained and updated to minimize the cross entropy loss function.

2. The finger recognition method according to claim 1, characterized in that: The steps to generate a capacitance data sequence include: For the collected capacitance data, the connected domain detection algorithm is used to extract the contact area blocks in each frame of capacitance data; Finger tracking is performed based on the spatiotemporal correlation of contact area blocks between multiple frames. Contact area blocks with intersections greater than a threshold in adjacent frames are classified as the same finger, and a capacitance data sequence of each finger is generated.

3. The finger recognition method according to claim 1, characterized in that: The steps to generate a capacitive image sequence include: Linearly mapping capacitance data in the capacitance data sequence into grayscale values ​​to generate a grayscale capacitance image for each frame; The grayscale capacitive image is preprocessed, and the image sequence is interpolated or sampled to obtain a fixed-length capacitive image sequence.

4. The finger recognition method according to claim 1, characterized in that: The encoder adopts but is not limited to a convolutional neural network, the decoder adopts but is not limited to a recurrent neural network, and the classifier adopts but is not limited to a fully connected layer.

5. A capacitive touch screen based finger recognition error correction method, which performs finger recognition based on the finger recognition method according to any one of claims 1 to 4, characterized in that: The following steps are involved: Divide the contact area blocks of the finger based on the capacitance data generated by the finger touching the capacitive screen, track the contact area blocks of continuous frames and identify their states; According to the state of the contact area block, determine whether the finger recognition trigger condition is met, and if so, perform finger recognition; During the finger recognition process, the contact area block sequence information of all fingers is used to determine whether the trigger condition for entering the error correction mode is met. If so, it is determined whether to enter the off-screen error correction mode or the non-off-screen error correction mode to perform finger recognition error correction. If the trigger condition for entering the error correction mode is not met, then determine one by one whether there is a contact area block sequence that meets the finger recognition trigger condition. If so, perform finger recognition and trigger a finger touch event. Otherwise, directly trigger a finger touch event. After triggering a finger touch event, record the finger touch event parameters and save the processing results of the current frame for use in subsequent frame processing.

6. The finger recognition error correction method according to claim 5, characterized in that: The states of the contact area block include: Just touched the screen: the current frame is the first frame, or there is no contact area block matching the contact area block of the current frame in the previous frame or multiple frames; Continue to maintain screen contact: There is a contact area block in the previous frame or multiple frames that matches the contact area block in the current frame; or Screen contact left: The contact area block from the previous frame or frames does not match any contact area block in the current frame.

7. The finger recognition error correction method according to claim 5, characterized in that: The finger recognition triggering conditions include: the number of data frames or the duration of the contact area block of a certain finger reaches a preset threshold; or the finger has left the screen and the finger recognition has not been completed; The trigger conditions for entering the error correction mode include: there are currently 5 or 10 finger contact area blocks that are touching the screen at the same time; or the finger touch intensity is greater than a preset threshold or the finger touch duration is greater than a preset threshold.

8. The finger recognition error correction method according to claim 5, characterized in that: The steps for finger recognition error correction include: 1) If the current mode is two-hand ten-finger error correction mode, divide the contact area blocks of the two hands and each of the five fingers according to the finger positions; 2) For each hand, the thumb, pinky and other fingers are distinguished according to the contact area block data of the five fingers, the steps including: calculating the maximum angle formed by the vertex of each finger and any two other vertices, sorting and selecting the two fingers with the smallest angle as the candidate fingers of the thumb and pinky; weighting and sorting the relationship between the candidate fingers and the other three fingers, selecting the one with the larger weighted sum as the thumb and the one with the smaller weighted sum as the pinky; 3) Based on the thumb and little finger distinction results obtained in step 2), the polar angles of other fingers are sorted with the thumb as the center. If the little finger is the smallest in the sorting result, it belongs to the left hand; if the little finger is the largest in the sorting result, it belongs to the right hand; 4) Combining the thumb and little finger distinction results obtained in step 2) and the left and right hand judgment results obtained in step 3), the categories of other fingers are determined by polar angle sorting, and finally the specific category of each finger of the left and right hands is obtained; the specific category is described in finger order, including: the finger order of the left hand is little finger, ring finger, middle finger, index finger, thumb, and the finger order of the right hand is thumb, index finger, middle finger, ring finger, little finger.

9. The finger recognition error correction method according to claim 5, characterized in that: Finger touch event parameters include: timestamp, finger contact position, contact area block capacitance data, whether it is in error correction mode, whether the finger has been recognized, and whether finger recognition is enabled; The processing result of the current frame includes: a contact area block sequence, contact area block capacitance data, and a finger recognition result.

10. A capacitive screen based finger recognition and error correction system, comprising the following modules: Capacitive sensor module: used to collect the capacitance data generated by the finger touching the capacitive screen in real time and generate the original capacitance data sequence; divide the finger contact area blocks based on the collected capacitance data and output the area block status of each frame; Display module: used to display the finger recognition results, including finger categories, touch events and related parameters; when entering the error correction mode, prompt the user to the error correction status, including off-screen error correction mode or non-off-screen error correction mode; Used to display error correction results and other interface elements to help users confirm or adjust the finger categories and touch events recognized by the system; Calculation module: used for finger recognition calculation, error correction mode calculation and dynamic event judgment; wherein, the finger recognition calculation includes: mapping the capacitance data sequence into a capacitance image sequence, and inputting the encoder of the neural network model frame by frame, extracting the spatial features of each frame of the capacitance image, and generating a hidden layer representation; inputting the hidden layer representation into the decoder, extracting the temporal features of the capacitance image sequence through time series modeling, and generating a hidden layer feature vector; inputting the hidden layer feature vector into the classifier, calculating the probability distribution of the finger category, and outputting the finger category with the highest probability as the finger recognition result; the error correction mode calculation includes: judging whether the finger recognition trigger condition is met according to the state of the contact area block, and triggering finger recognition; judging whether the trigger condition for entering the error correction mode is met, further distinguishing the off-screen error correction mode or the on-screen error correction mode, and recalculating the finger category and touch event in the error correction mode; the dynamic event judgment includes: if the finger touch event does not need error correction, then judging whether there is a contact area block sequence that meets the finger recognition trigger condition, and directly triggering the corresponding event; Storage module: used to save the execution program, including the operating logic of finger recognition and error correction; store the neural network model of finger recognition and its training parameters; save finger touch event parameters, recognition results, error correction results and finger contact area block information.

Images