A touch screen false touch point detection and defense method based on spatiotemporal network

Abstract

The present invention discloses a touch screen false touch point detection and defense method based on a spatiotemporal network, which belongs to the field of vulnerability detection. The method comprises: reading the capacitance value data of a capacitive touch screen, wherein the capacitance value data is a two-dimensional tensor; preprocessing the capacitance value data; performing binary threshold segmentation on the preprocessed capacitance value data to filter out capacitance abnormal value areas; performing refinement processing on the capacitance abnormal value areas to obtain a refined two-dimensional tensor; constructing a false touch point detection model, sequentially extracting the spatial features and temporal features of the refined two-dimensional tensor, and converting the final features into the probability that the capacitive touch screen is currently under attack; using normal touch samples and attacked samples to train the false touch point detection model, using the trained model to predict the probability that the capacitive touch screen is currently under attack, and taking defense measures based on the prediction results. The present invention is of great significance to ensuring the security of capacitive touch screen terminals.

Description

Technical Field

[0001] The present invention belongs to the field of vulnerability detection, and in particular relates to a touch screen false touch point detection and defense method based on a spatiotemporal network. Background Art

[0002] With the rapid adoption of smart devices, capacitive touchscreens, as a tool for human-computer interaction, have gained widespread adoption in smart devices due to their simplicity, convenience, durability, and compact size. They have become the most widely used and common touchscreen terminal. However, capacitive touchscreens also pose security risks, potentially exposing sensitive user information. By attacking a capacitive touchscreen, an attacker could control the phone to perform a range of actions, such as opening banking apps and sending private messages, compromising the user's privacy and financial security. Therefore, the security of capacitive touchscreens is paramount.

[0003] The "ghost hand effect" refers to a phenomenon in which random touch points are generated on the capacitive touch screen of a mobile phone without the user actively operating the phone. This can cause a series of erroneous operations on the phone, such as opening desktop applications, chat software, making calls, making payments, and transmitting data. Existing research has made these "ghost hand" actions controllable. If exploited by criminals, this technology would directly affect the privacy and property security of mobile phone users. Currently, many related studies focus on how to generate "ghost hand attack" signals, but lack methods for detecting and defending against such attacks. Therefore, developing a method for detecting and defending against ghost hand attacks is of great significance.

[0004] The principle of existing ghost hand attacks based on charging line interference is to change the capacitance value in the direction of the conductive electrode (driving electrode or sensing electrode), thereby injecting a row or column of false contacts in the electrode direction. Compared with the single-point capacitance value change pattern under normal touch operation, this type of attack will cause the capacitance value of the capacitive screen to suddenly change in the horizontal or vertical direction. In normal use, the contacts are mostly isolated from each other. Even during operations such as sliding, the contacts are connected in time-sharing rather than simultaneously. Utilizing the capacitance distribution characteristics of the touch screen under normal use and ghost hand attacks, the present invention proposes a touch screen false touch point detection and defense method based on spatiotemporal network. Summary of the Invention

[0005] The purpose of this invention is to provide a method for detecting and preventing false touch points on touch screens based on a spatiotemporal network. This method addresses the fact that many existing studies focus on generating "ghost hand attack" signals, but fail to address the problem of detecting and preventing such attacks. The key technology of this invention is to extract the spatiotemporal characteristics of capacitance changes in capacitive touch screens. For such two-dimensional data, this invention uses a CNN to extract spatial features and a ConvLSTM to extract temporal features, which are then further analyzed.

[0006] The present invention adopts the following technical solutions to solve the above problems:

[0007] A method for detecting and preventing false touch points on a touch screen based on a spatiotemporal network, comprising the following steps:

[0008] Step 1: reading capacitance data of the capacitive touch screen, wherein the capacitance data is a two-dimensional tensor;

[0009] Step 2: preprocessing the capacitance value data obtained in step 1;

[0010] Step 3: perform binary threshold segmentation on the pre-processed capacitance value data to filter out capacitance abnormal value areas;

[0011] Step 4: Refine the capacitance abnormal value region to obtain a refined two-dimensional tensor;

[0012] Step 5: Construct a false touch point detection model, including a spatial feature extraction network, a temporal feature extraction network, and a fully connected layer. The spatial feature extraction network is used to extract spatial features of the refined two-dimensional tensor, the temporal feature extraction network is used to extract temporal features from the spatial feature data as the final features of the two-dimensional tensor, and the fully connected layer is used to convert the final features of the two-dimensional tensor into the probability that the capacitive touch screen is currently under attack.

[0013] Step 6: Use normal touch samples and attacked samples to train a false touch point detection model, use the trained model to predict the probability of the capacitive touch screen currently being attacked, and take defensive measures based on the predicted results.

[0014] Furthermore, the preprocessing includes filtering and normalization processing.

[0015] Furthermore, the step three is specifically as follows:

[0016] 3.1) According to the capacitance values ​​in the two-dimensional tensor, the capacitance values ​​are evenly divided into L levels, and the number of capacitance values ​​falling in level i is recorded as N i ;

[0017] Calculate the probability of occurrence and average capacitance value of capacitance values ​​from level 0 to level k:

[0018]

[0019]

[0020] Where w(k) represents the probability of occurrence of capacitance values ​​from level 0 to level k, w(L-1)=1; u(k) represents the average capacitance value from level 0 to level k, and the total average capacitance value is recorded as u T =u(L-1);

[0021] 3.2) According to the preset threshold t1, the two-dimensional tensor is divided into two categories C j ,j∈{1,2},C1∈[t0,t1],C2∈[t1,t2], where t0=0,t2=L-1, then the jth class C j The probability of occurrence w j , average capacitance value u j and variance The calculation formula is:

[0022] w j =w(t j )-w(t j-1 )

[0023]

[0024]

[0025] j∈{1,2}

[0026] 3.3) Calculate intra-class variance

[0027]

[0028] Calculate between-class variance

[0029]

[0030] 3.4) The threshold t1 that minimizes the intra-class variance or maximizes the inter-class variance is used as the optimal threshold to perform binary segmentation on the two-dimensional tensor.

[0031] Furthermore, the step 4 is specifically as follows:

[0032] 4.1) The capacitance outlier region is marked as the target point and marked as 1, and the rest are marked as background points and marked as 0; the boundary points are marked from the target points. For all boundary points P1 in the capacitance outlier region, the clockwise 8 neighbors of the boundary point are marked as P2, P3, ..., P9, where P2 is above P1;

[0033] 4.2) Loop through all boundary points and add deletion marks to those boundary points that meet the following conditions:

[0034] (a) 2≤N(P1)≤6, where N(P1) is the number of non-zero neighbors of P1;

[0035] (b) S(P1) = 1, where S(P1) is the number of times the values ​​of these points change from 0 to 1 when the order is P2, P3, ..., P9;

[0036] (c) P2*P4*P6=0;

[0037] (d) P4*P6*P8=0;

[0038] 4.3) Loop through all boundary points and add deletion marks to those boundary points that meet the following conditions:

[0039] (a)2≤N(P1)≤6;

[0040] (b) S(P1) = 1;

[0041] (c) P2*P4*P8=0;

[0042] (d) P2*P6*P8=0;

[0043] 4.4) Set the boundary points marked with deletion to 0, converting them into background points, and use the retained target points as the updated capacitance outlier region;

[0044] 4.5) Repeat steps 4.1) to 4.4) until the capacitance outlier region is no longer updated, and use the final capacitance outlier region as the capacitance skeleton topology to obtain a refined two-dimensional tensor.

[0045] Furthermore, the spatial feature extraction network is implemented using a multi-layer convolutional neural network consisting of n convolutional blocks connected in series. The convolution kernel of the convolutional neural network corresponds to the direction of the drive electrodes TX or the sensing electrodes RX in the capacitive touch screen and is selected from one or more of the following shapes: a "|", a "s", a "cross", and an "X". A rectified linear unit (ReLU) is used as the activation function of the CNN component, and maximum pooling is used for feature dimensionality reduction in each convolutional layer.

[0046] Furthermore, the temporal feature extraction network includes m cascaded ConvLSTM layers and a spatial pyramid pooling layer, the input of the m cascaded ConvLSTM layers is a three-dimensional tensor composed of several two-dimensional tensors, the intermediate results generated by internal calculations in the ConvLSTM layer include cell states, hidden variables, and each intermediate state output is a three-dimensional tensor, and finally a high-level feature map is output;

[0047] The spatial pyramid pooling layer divides the high-level feature map into n layers. In the i-th layer, the feature map is divided into 2 in both x and y directions. i That is, the i-th layer divides the feature map into 2 2i After the segmentation is completed, the maximum value of each block is taken as one dimension in the feature vector, and finally an output is The dimensional feature vector is used as the final feature of each 2D tensor.

[0048] The above feature vector is sent to the fully connected layer, and finally the probability of the current capacitive touch screen being attacked is output.

[0049] Furthermore, the number of layers of the spatial pyramid pooling layer is 2-4, preferably 3.

[0050] Furthermore, the defense measures taken based on the prediction results are: if the current predicted probability of being attacked is greater than p1, the contact detection mechanism is disabled and the touch screen operation is not responded to; if the current predicted probability of being attacked is greater than p2, a pop-up prompt is issued on the screen, p1>p2.

[0051] The present invention's touch screen false touch point detection and defense method based on spatiotemporal network has a reasonable design, convenient operation, strong portability, good flexibility, reliable detection results, and a simple defense mechanism. The main benefits include:

[0052] (1) The present invention addresses the ghost hand effect of capacitive screens, breaking through the existing research focus on how to generate "ghost hand attack" signals, and proposes a method for detecting and defending against ghost hand attacks, which is of great significance to ensuring the security of capacitive touch screen terminals.

[0053] (2) The present invention mainly relies on algorithms and does not require any additional hardware. If large-scale deployment is required, it only needs to be written into the bottom layer of the mobile phone system. It is easy to operate and has strong portability.

[0054] (3) The present invention regards the capacitance value matrix as a picture and preprocesses it to make its features more obvious. The preprocessed two-dimensional tensor is input into the convolutional neural network and convolved with the convolution kernel. The convolution kernel can be flexibly modified so that the convolutional neural network can extract different features, which effectively improves the flexibility of the present invention. In addition, many parameters in the algorithm used in the present invention can be flexibly modified according to actual needs, so that detection and defense can achieve better results. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a schematic diagram of the overall flow chart of the touch screen false touch point detection and defense method based on the spatiotemporal network of the present invention;

[0056] Figure 2It is a schematic diagram of 8 neighborhoods in the present invention;

[0057] Figure 3 It is a schematic diagram of the activation function in the present invention;

[0058] Figure 4 This is an example diagram of capacitance change before and after the attack and data preprocessing in the present invention;

[0059] Figure 5 It is a spatiotemporal network structure diagram in the present invention;

[0060] Figure 6 It is a flowchart of the specific defense process in the present invention;

[0061] Figure 7 It is a schematic diagram of the reference convolution kernel in the present invention. DETAILED DESCRIPTION

[0062] The present invention is further described below with reference to the accompanying drawings and examples, so that the present invention is more comprehensive and complete and the concept of the exemplary embodiments is fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0063] In addition, the drawings are merely schematic illustrations of the present invention, and some of the block diagrams shown in the drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0064] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to actual circumstances.

[0065] The present invention utilizes the characteristic that a capacitive screen attacked by a ghost hand often experiences a sudden change in capacitance in the horizontal or vertical direction. Combining tools such as filtering algorithms, refinement algorithms, convolutional neural networks (CNNs), and convolutional long short-term memory artificial neural networks (ConvLSTMs), the present invention proposes a touch screen false touch point detection and defense method based on a spatiotemporal network. This method realizes automated ghost hand attack detection and defense, can effectively detect whether a capacitive screen terminal is under a ghost hand attack, and can take efficient defense measures.

[0066] Figure 1 The overall process of this exemplary embodiment is shown. Figure 1 As shown, a touch screen false touch point detection and defense method based on spatiotemporal network includes the following steps:

[0067] Step 1: directly obtain the capacitance data of the capacitive touch screen of the mobile phone and obtain a 17*33 two-dimensional tensor of the contact capacitance value. The two-dimensional tensor of the capacitance value when not attacked is as follows: Figure 4 As shown in (a), the two-dimensional tensor of the capacitance value when the 12th driver terminal TX is attacked is as follows Figure 4 As shown in (b), after the capacitive screen is attacked, a significant vertical capacitance mutation occurs. During the training phase, it is necessary to obtain capacitance data of the capacitive touch screen that has been attacked and that has not been attacked.

[0068] Step 2: Preprocess the capacitance value obtained in step 1, including filtering to reduce noise and interference, and normalization.

[0069] 2.1) Clipped Average Filtering: To reduce the interference of random and periodic errors, clipped average filtering is used to remove random interference errors. The clipped average filtering algorithm combines clipping and recursive average filtering. Each new data sample is first clipped before being sent to the queue for recursive average filtering. It combines the advantages of both filtering algorithms while overcoming their shortcomings. For occasional pulse interference, it can eliminate sampling value deviations caused by pulse interference. It also effectively suppresses periodic interference, improves data smoothness, and meets the requirements for measurement speed.

[0070] 2.2) (0,1) Normalization: To facilitate subsequent data processing and speed up network learning, the filtered capacitance data is normalized. Considering that the capacitance scan values ​​of the same object at consecutive moments are normalized, the difference between the capacitance extreme values ​​before and after the moment is not large, the (0,1) normalization method is adopted for normalization, and a series of 17*33 two-dimensional tensors with values ​​between (0,1) are output. (0,1) normalization is performed according to formula (1):

[0071]

[0072] Among them, C(x,y) represents the original capacitance value of the mobile phone screen scanning point (x,y), C max Indicates the maximum capacitance of all scanning points in the frame sampling, C min Indicates the minimum capacitance value among all scanning points in the frame sampling, and C′(x,y) represents the normalized capacitance value at the scanning point (x,y) on the mobile phone screen.

[0073] Step 3: Binarization threshold segmentation: Binarization threshold segmentation is performed on the obtained two-dimensional tensor.

[0074] In this embodiment, the Otsu method is used to determine the optimal threshold. The criterion for determining the optimal threshold by the Otsu method is to minimize the intra-class variance of each class after threshold segmentation, and at this time, the inter-class variance after threshold segmentation is also the largest. The result of Otsu method threshold segmentation is as follows: Figure 4 As shown in (c), it can be seen that the Otsu method can successfully extract the distribution of abnormal capacitance values.

[0075] 3.1) The total number of capacitance values ​​in the obtained two-dimensional tensor is N = 561, and the capacitance values ​​are evenly divided into L = 256 levels, and the number of capacitance values ​​falling in level i is recorded as N i Let w(k) and u(k) represent the probability of occurrence of capacitance values ​​from level 0 to level k and the average capacitance value, respectively. The calculation formula is:

[0076]

[0077]

[0078] The probability of the capacitance value from level 0 to level L-1 is 1, which is recorded as w(L-1)=1, and the total average capacitance value is recorded as u T =u(L-1).

[0079] 3.2) Use a threshold t1 to divide the two-dimensional tensor into two categories C j ,j∈{1,2},C1∈[t0,t1],C2∈[t1,t2], where t0=0, t1, t2=L-1, then C j The probability of occurrence w j , average capacitance value u j and variance The calculation formula is:

[0080] w j =w(t j )-w(t j-1 )

[0081]

[0082]

[0083] j∈{1,2}

[0084] 3.3) From this we can get the intra-class variance:

[0085]

[0086] The between-class variance of each class is:

[0087]

[0088] 3.4) The threshold t1 that minimizes the intra-class variance or maximizes the inter-class variance is used as the optimal threshold for thresholding, and the two-dimensional tensor is binarized and segmented. Figure 4 The segmentation threshold in (c) is 0.59, and all points with normalized capacitance values ​​greater than or equal to 0.59 are set to 1, and points with values ​​less than 0.59 are set to 0.

[0089] Step 4: To highlight the shape characteristics and reduce the amount of redundant information, facilitate the subsequent neural network operation, and make the connectivity features more obvious, the binarized two-dimensional tensor is refined to obtain the topological structure of the mobile phone capacitor skeleton. The refinement result is as follows Figure 4 As shown in (d), it can be seen that the refinement algorithm successfully extracts the connectivity features of the capacitance distribution.

[0090] 4.1) Let the known target points be marked as 1 and the background points be marked as 0. A boundary point is a point that is marked as 1 and has at least one point marked as 0 in its 8-connected neighborhood.

[0091] like Figure 2 As shown in the figure, consider the 8-neighborhood centered on the boundary point, let P1 be the center point, and mark the 8 points in its neighborhood counterclockwise around the center point as P2, P3, ..., P9, where P2 is above P1.

[0092] 4.2) Loop through all boundary points, set the boundary point as P1, and mark the points that meet the following conditions as deleted.

[0093] (a) 2≤N(P1)≤6, where N(P1) is the number of non-zero neighbors of P1;

[0094] (b) S(P1) = 1, where S(P1) is the number of times the values ​​of these points change from 0 to 1 when the order is P2, P3, ..., P9;

[0095] (c) P2*P4*P6=0;

[0096] (d) P4*P6*P8=0;

[0097] 4.3) Similar to 4.2, conditions (a) and (b) are exactly the same, and only conditions (c) and (d) are different. Point P1 that meets the following conditions is marked for deletion. The conditions are as follows:

[0098] (a)2≤N(P1)≤6;

[0099] (b) S(P1) = 1;

[0100] (c) P2*P4*P8=0;

[0101] (d) P2*P6*P8=0;

[0102] 4.4) Set the points marked for deletion in steps 4.2) and 4.3) to 0 (turning them into background points);

[0103] 4.5) Repeat steps 4.2) to 4.4) until no points are marked for deletion; the output result is the refined topological structure of the mobile phone capacitor skeleton.

[0104] Step 5. The data processed through the above steps is a series of two-dimensional tensors. Although LSTM has been proven to be able to handle long-distance dependency problems well, the input of the network is a one-dimensional vector. If the LSTM network is applied directly, the two-dimensional tensor needs to be flattened into a one-dimensional vector, which will undoubtedly result in a large loss of spatial information. In order to solve this problem, a ConvLSTM network is used, which can receive data as input of two-dimensional tensors, thereby better preserving the spatial characteristics of the capacitance distribution. At the same time, in order to avoid the input of ConvLSTM being too large, a CNN network is introduced before ConvLSTM to extract features and reduce the amount of data. Then a CNN-ConvLSTM model is built, and the preprocessed training set data is divided into several samples at equal intervals, input into the input end of the machine learning model, randomly initialize the weights, and use the gradient descent algorithm to optimize the model parameters. The model is built using the TensorFlow and Keras frameworks. The specific steps are:

[0105] 5.1) Design the convolution kernel shape: First, according to the direction of the driving electrode TX or the induction motor RX of the device touch screen, construct a narrow and long convolution kernel along these two directions. Figure 7 As shown, for example, if the TX of a device arranged vertically is vulnerable to attack, a "|"-shaped convolution kernel can be constructed; if the RX of a device arranged horizontally is vulnerable to attack, a "I"-shaped convolution kernel can be constructed; at the same time, commonly used "cross" and "X"-shaped convolution kernels can be added.

[0106] 5.2) Construct convolutional blocks:

[0107] 5.2.1) Convolutional layer: Use the following formula (2) to calculate the feature map of each layer, and establish multiple convolutions. The weights W and bias b of the network parameters of each layer are obtained by gradient descent training. Let the position (x i ,y j ) has a value of ij ,but

[0108]

[0109] Among them, f(.) is the activation function; b is the bias of the feature map, w mn is the value corresponding to position x i+m,j+nThe convolution kernel unit, M and N are the height and width of the convolution kernel respectively;

[0110] 5.2.2) Activation function: The linear rectification function (ReLU, as shown in formula (3)) is used as the activation function of the CNN part:

[0111] f(x)=max(0,x) (3)

[0112] Since a series of images need to be processed, the above two steps can be implemented using tensorflow.keras.conv2D, and the activation parameter is selected as 'relu';

[0113] 5.2.3) Pooling layer: Max pooling is used for feature dimensionality reduction in each convolutional layer. This can be implemented using tensorflow.keras.MaxPooling2D.

[0114] In this embodiment, a three-dimensional convolutional neural network (3D-CNN) can be used to speed up processing, but care must be taken to ensure that the size of the convolution kernel and pooling layer in the time dimension is 1 to avoid a reduction in the number of frames. For example, the size of the convolution kernel is 5×5×1, two convolution layers are used, and the number of convolution kernels is set to 16 and 32 respectively; the size of all pooling layers is set to 3×3×1; and all layers use the ReLU function as the nonlinear activation function. The convolution kernel and fully connected weights are initialized with truncated normal distribution random numbers with a standard deviation of 0.1; the Adam optimization algorithm is used to minimize the cross entropy loss calculation, with a learning rate of 0.0001, a batch size of 32, and 80 iterations.

[0115] 5.3) Constructing ConvLSTM layer: The main structure of ConvLSTM layer is consistent with the traditional LSTM. The main difference is that the convolution structure is used to transform the input data from a one-dimensional vector to a three-dimensional tensor, and the cell state C t , hidden variable h t , each intermediate state output f t ,i t , C t , p t All become three-dimensional tensors.

[0116] Its network structure can be expressed by formula (4):

[0117]

[0118]

[0119] Among them, x t is the t-th capacitance value two-dimensional tensor of the input, t is the batch size, C t is the cell state, ht is the hidden state, i t 、f t 、p t 、C t Respectively represent the state quantities corresponding to the input gate, memory gate, output gate, and core gate of ConvLSTM at time t, where time t corresponds to the t-th capacitance value two-dimensional tensor. σ is the Sigmoid function, W i 、W f 、W p 、W c Represent the weight transfer matrices corresponding to the input gate, memory gate, output gate, and core gate, respectively. U i 、U f 、U p 、U c Represents the hidden layer variable h of the input gate, memory gate, output gate and core gate at time t-1 respectively t-1 The corresponding weight transfer matrix, b i , b f , b p , b k Represent the bias vectors corresponding to the input gate, memory gate, output gate and core gate respectively;

[0120] The ConvLSTM layer can be implemented using tensorflow.keras.convLSTM2D.

[0121] 5.4) If Figure 5 As shown, establish the network architecture:

[0122] 5.4.1) For the preprocessed two-dimensional data sequence, spatial feature extraction is performed using a spatial feature extraction network consisting of two convolutional blocks in series. This can be achieved using keras.models.Sequential in series with conv2D and MaxPooling2D.

[0123] 5.4.2) Input the spatial feature extraction results (2D data) into two cascaded ConvLSTM layers to extract temporal features and output a high-level feature map. This can be achieved by concatenating two convLSTM2D layers using keras.models.Sequential.

[0124] 5.4.3) Spatial Pyramid Pooling: Perform spatial pyramid pooling on the high-level feature map output by ConvLSTM, that is, divide the feature map into 3 layers, and the i-th layer divides the feature map into 2 layers in both x and y directions. i That is, the i-th layer divides the feature map into 2 2i After the segmentation is completed, the maximum value of each block is taken as one dimension in the feature vector, and finally a -dimensional feature vector;

[0125] In this embodiment, two cascaded ConvLSTM layers and a spatial pyramid pooling layer constitute a temporal feature extraction network.

[0126] 5.4.4) Fully Connected Layer: The feature vector output by spatial pyramid pooling is fed into the fully connected layer, which finally outputs the probability of the current capacitive touch screen being attacked. The activation function uses the softmax function, which can be implemented using tensorflow.keras.Dense(1, activation = 'softmax'). This embodiment uses two fully connected layers.

[0127] 5.5) Network Training: Using capacitance data from unattacked and attacked capacitive touchscreens as positive and negative sample data, supervised learning and gradient descent are used to train the parameters of the spatial feature extraction network, temporal feature extraction network, and fully connected layer. Batch regularization and 5-fold cross-validation are used during training: each class of samples is randomly divided into five equal parts beforehand, and then four samples from different classes are randomly selected to form a training set, with the remaining one set serving as a test set. The Adam optimizer is used to minimize the cross-entropy loss. During model training, the model outputs probability values ​​in the range (0, 1).

[0128] Among them, batch normalization can be implemented using tensorflow.keras.BatchNormalization(); 5-fold cross validation can be implemented using sklearn.model_selection.StratifiedKFold(n_splits=5,shuffle=True,random_state=seed); Adam optimizer can be implemented in Keras using model.compile(loss='categorical_crossentropy',optimizer=Adam(learning_rate=1e-4),metrics=['accuracy']).

[0129] 5.6) Performance evaluation: Three indicators were used for evaluation, including accuracy (ACC), sensitivity (SEN) and specificity (SPE), as shown in the following formulas (5)(6)(7);

[0130]

[0131]

[0132]

[0133] Among them, TP is a true positive case, FP is a false positive case, TN is a true negative case, and FN is a false negative case.

[0134] The above three performance indicators can be added to model.metrics_names and model.evaluate(x,y,verbose=0) in Keras to achieve performance evaluation.

[0135] Step 6, such as Figure 6 As shown, for the capacitive touch screen device to be defended, steps 1 to 4 are used to pre-process the capacitance data, and the trained model is used to perform security detection. According to the detection results, corresponding defense measures are taken: if the probability of being attacked is greater than 80%, the contact detection mechanism is disabled. Even if the capacitance value changes, the contact is not detected and the touch screen operation is not responded. If the probability of being attacked is greater than 50%, the mobile phone sends a pop-up prompt to prompt the user to pay attention and check.

[0136] The above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above examples, and many variations are possible. All variations that can be directly derived or imagined by a person skilled in the art from the disclosure of the present invention should be considered to be within the scope of protection of the present invention.

Claims

1. A touch screen false touch point detection and defense method based on spatiotemporal network, characterized in that: The following steps are involved: Step 1: reading capacitance data of the capacitive touch screen, wherein the capacitance data is a two-dimensional tensor; Step 2: preprocessing the capacitance value data obtained in step 1; Step 3: perform binary threshold segmentation on the pre-processed capacitance value data to filter out capacitance abnormal value areas; Step 4: Refine the capacitance abnormal value region to obtain a refined two-dimensional tensor; The step four is specifically as follows: 4.1) The capacitance outlier region is marked as the target point and marked as 1, and the rest are marked as background points and marked as 0; the boundary points are marked from the target points. For all boundary points P1 in the capacitance outlier region, the clockwise 8 neighbors of the boundary point are marked as P2, P3, ..., P9, where P2 is above P1; 4.2) Loop through all boundary points and add deletion marks to those boundary points that meet the following conditions: (a) 2≤N(P1)≤6, where N(P1) is the number of non-zero neighbors of P1; (b) S(P1) = 1, where S(P1) is the number of times the values ​​of these points change from 0 to 1 when the order is P2, P3, ..., P9; (c) P2*P4*P6=0; (d) P4*P6*P8=0; 4.3) Loop through all boundary points and add deletion marks to those boundary points that meet the following conditions: (a)2≤N(P1)≤6; (b) S(P1) = 1; (c) P2*P4*P8=0; (d) P2*P6*P8=0; 4.4) Set the boundary points marked with deletion to 0, converting them into background points, and use the retained target points as the updated capacitance outlier region; 4.5) Repeat steps 4.1) to 4.4) until the capacitance outlier region is no longer updated, and use the final capacitance outlier region as the capacitance skeleton topology to obtain a refined two-dimensional tensor; Step 5: Construct a false touch point detection model, including a spatial feature extraction network, a temporal feature extraction network, and a fully connected layer. The spatial feature extraction network is used to extract spatial features of the refined two-dimensional tensor, the temporal feature extraction network is used to extract temporal features from the spatial feature data as the final features of the two-dimensional tensor, and the fully connected layer is used to convert the final features of the two-dimensional tensor into the probability that the capacitive touch screen is currently under attack. Step 6: Use normal touch samples and attacked samples to train a false touch point detection model, use the trained model to predict the probability of the capacitive touch screen currently being attacked, and take defensive measures based on the predicted results.

2. The touch screen false touch point detection and defense method based on spatiotemporal network according to claim 1 is characterized in that: The preprocessing includes filtering and normalization.

3. The touch screen false touch point detection and defense method based on spatiotemporal network according to claim 1, characterized in that: The step three is specifically as follows: 3.1) According to the capacitance values ​​in the two-dimensional tensor, the capacitance values ​​are evenly divided into L levels, and the number of capacitance values ​​falling in level i is recorded as N i ; Calculate the probability of occurrence and average capacitance value of capacitance values ​​from level 0 to level k: Where w(k) represents the probability of occurrence of capacitance values ​​from level 0 to level k, w(L-1)=1; u(k) represents the average capacitance value from level 0 to level k, and the total average capacitance value is recorded as u T =u(L-1); 3.2) According to the preset threshold t1, the two-dimensional tensor is divided into two categories C j ,j∈{1,2},C1∈[t0,t1],C2∈[t1,t2], where t0=0,t2=L-1, then the jth class C j The probability of occurrence w j , average capacitance value u j and variance The calculation formula is: w j =w(t j )-w(t j-1 ) j∈{1,2} 3.3) Calculate intra-class variance Calculate between-class variance 3.4) The threshold t1 that minimizes the intra-class variance or maximizes the inter-class variance is used as the optimal threshold to perform binary segmentation on the two-dimensional tensor.

4. The touch screen false touch point detection and defense method based on spatiotemporal network according to claim 1, characterized in that: The spatial feature extraction network is implemented using a multi-layer convolutional neural network. The shape of the convolution kernel of the convolutional neural network corresponds to the direction of the driving electrode TX or the sensing electrode RX in the capacitive touch screen, and is selected from one or more of the following shapes: "|", "I", "X", and "X".

5. The touch screen false touch point detection and defense method based on spatiotemporal network according to claim 4, characterized in that: Each layer in the multi-layer convolutional neural network consists of a convolution layer and a pooling layer, and the convolution layer adopts the ReLU activation function.

6. The touch screen false touch point detection and defense method based on spatiotemporal network according to claim 1, characterized in that: The temporal feature extraction network includes m cascaded ConvLSTM layers and a spatial pyramid pooling layer. The input of the m cascaded ConvLSTM layers is a three-dimensional tensor composed of several two-dimensional tensors. The intermediate results generated by internal calculations in the ConvLSTM layer include cell states, hidden variables, and each intermediate state output is a three-dimensional tensor, and finally a high-level feature map is output; The spatial pyramid pooling layer divides the high-level feature map into n layers. In the i-th layer, the feature map is divided into 2 in both x and y directions. i That is, the i-th layer divides the feature map into 2 2i After the segmentation is completed, the maximum value of each block is taken as one dimension in the feature vector, and finally an output is The dimensional feature vector is used as the final feature of each 2D tensor.

7. The touch screen false touch point detection and defense method based on spatiotemporal network according to claim 6, characterized in that: The number of layers of the spatial pyramid pooling layer is 2-4.

8. The touch screen false touch point detection and defense method based on spatiotemporal network according to claim 1, characterized in that: The defense measures taken based on the prediction results are: if the current predicted probability of attack is greater than p1, the contact detection mechanism is disabled and the touch screen operation is not responded; if the current predicted probability of attack is greater than p2, a pop-up prompt is issued on the screen, p1>p2.

Images