Touch position prediction method and device, touch equipment, electronic equipment and storage medium
Through the self-supervised learning offset position prediction network, combined with the similarity training of multi-frame touch sensing image samples, the problem of insufficient accuracy and speed of touch position prediction in the prior art is solved, and the sub-pixel precision touch position prediction is achieved, ensuring a smooth user interaction experience.
Patent Information
- Application Number
- CN202411696566.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-25
- Publication Date
- 2025-08-12
AI Technical Summary
The existing touch position prediction scheme cannot accurately detect the pressing state and position of the stylus in real time, especially in sub-pixel accuracy, which cannot meet the smooth user interaction experience.
Using a self-supervised learning offset position prediction network, through the similarity training of multi-frame touch sensing image samples, the offset position of the touch object is predicted, and combined with the first touch position is fused to achieve sub-pixel accuracy touch position determination.
It realizes that the touch position is accurately predicted through self-supervised without sub-pixel labeling data, which meets the requirements of fast response and sub-pixel accuracy, and ensures natural and accurate feedback of stylus operation.
Smart Images
Figure CN120469592A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of touch devices, and in particular to a touch position prediction method, apparatus, touch device, electronic device, and storage medium. Background Art
[0002] A touch screen stylus is a common, low-cost touch and writing interaction technology. Typically, a touch sensor array is placed beneath the touch screen. Based on the pixel size of the touch sensor array and the physical dimensions of the screen, a touch-sensing image with a resolution similar to that of pixels (e.g., 16×24 pixels) can be generated.
[0003] The touch sensing image can be used to identify the touch position of a stylus. To ensure a smooth user interaction experience, the stylus's pressing state and precise location must be detected in real time at a high rate (for example, 200 frames per second). However, existing touch position prediction solutions cannot accurately locate the touch position. Summary of the Invention
[0004] In view of the above problems, embodiments of the present application provide a touch position prediction method, apparatus, touch device, electronic device, and storage medium to overcome the above problems or at least partially solve the above problems.
[0005] In a first aspect of an embodiment of the present application, a touch position prediction method is disclosed, the method comprising: Acquire a touch sensing image; determining a first touch position of the touch object on the touch screen according to the touch sensing image; Predicting an offset position based on a target image region in the touch sensing image using a pre-trained offset position prediction network, wherein the offset position prediction network learns similarities between offset positions of multiple frames of touch sensing image samples; The touch position of the touch object on the touch screen is determined according to the first touch position and the offset position.
[0006] Optionally, it also includes: A target image area is extracted from the touch-sensing image according to the first touch position, where the target image area includes pixels corresponding to the first touch position in the touch-sensing image.
[0007] Optionally, extracting a target image area from the touch-sensing image according to the first touch position includes: A target image area is extracted from the touch-sensing image with the pixel point corresponding to the first touch position in the touch-sensing image as the center.
[0008] Optionally, it also includes: Acquire a training data set, wherein the training data set includes a plurality of touch sensing image samples collected at a target time interval; Every two adjacent frames of sensor image samples in the training data set are used as a training sample, and self-supervised training is performed on the neural network to obtain the offset position prediction network.
[0009] Optionally, taking every two adjacent frames of sensor image samples in the training data set as a training sample, performing self-supervised training on a neural network to obtain the offset position prediction network includes: Inputting a target image area sample from a first frame of touch-sensing image samples in the training samples into the neural network to obtain a first offset position prediction result, and inputting a target image area sample from a second frame of touch-sensing image samples in the training samples into the neural network to obtain a second offset position prediction result, wherein the target image area sample includes pixel points corresponding to the sample touch position in the touch-sensing image sample; With the goal of maximizing the similarity between the first offset position prediction result and the second offset position prediction result, self-supervised training is performed on the neural network to obtain the offset position prediction network.
[0010] Optionally, the resolution of the touch sensing image is a first resolution corresponding to the touch sensing element array of the touch screen; Determining a first touch position of the touch object on the touch screen according to the touch sensing image includes: determining, according to the touch-sensing image, a first touch position of the touch object on the touch screen at an accuracy corresponding to the first resolution; Predicting an offset position based on a target image area in the touch sensing image using a pre-trained offset position prediction network includes: At the accuracy corresponding to the second resolution, the offset position is predicted according to the target image area through a pre-trained offset position prediction network. The second resolution is the resolution of the display pixels of the touch screen, and the second resolution is greater than the first resolution.
[0011] Optionally, the first touch position includes a first coordinate position and a second coordinate position, and the offset position includes a first coordinate offset position and a second coordinate offset position; Determining a touch position of the touch object on the touch screen according to the first touch position and the offset position includes: Merging the first coordinate position and the first coordinate offset position to obtain a first coordinate, and merging the second coordinate position and the second coordinate offset position to obtain a second coordinate; A touch position of the touch object on the touch screen is determined according to the first coordinate and the second coordinate.
[0012] According to a second aspect of the embodiments of the present application, a touch device is disclosed, comprising: A touch screen comprising an array of touch sensing elements, wherein when a touch object contacts the touch screen, a sensing signal of the touch sensing element array changes, and a touch sensing image is generated according to the sensing signal; The processor is configured to execute the steps of the touch position prediction method described in the first aspect of the embodiment of the present application to determine the touch position of the touch object on the touch screen.
[0013] According to a third aspect of the present application, a touch position prediction device is disclosed, comprising: A first acquisition module, configured to acquire a touch sensing image; a first determining module, configured to determine a first touch position of the touch object on the touch screen according to the touch sensing image; a first prediction module, configured to predict an offset position based on a target image area in the touch sensing image using a pre-trained offset position prediction network, wherein the offset position prediction network learns similarities between offset positions of multiple frames of touch sensing image samples; The second determining module is configured to determine a touch position of the touch object on the touch screen according to the first touch position and the offset position.
[0014] According to a fourth aspect of an embodiment of the present application, an electronic device is disclosed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the touch position prediction method described in the first aspect of the embodiment of the present application are implemented.
[0015] According to a fifth aspect of the embodiments of the present application, a computer-readable storage medium is disclosed, on which a computer program is stored. When the computer program is executed by a processor, the steps of the touch position prediction method described in the first aspect of the embodiments of the present application are implemented.
[0016] According to a sixth aspect of the embodiments of the present application, a computer program product is disclosed, including a computer program. When the computer program is executed by a processor, the steps of the touch position prediction method according to the first aspect of the embodiments of the present application are implemented.
[0017] The embodiments of the present application include the following advantages: In an embodiment of the present application, a first touch position of a touch object on a touch screen is determined based on a touch sensing image; and an offset position is predicted based on a target image area in the touch sensing image using a pre-trained offset position prediction network. Because the offset position prediction network learns the similarity between the offset positions of multiple frames of touch sensing image samples, the offset position prediction network has good position prediction performance. Based on the offset position prediction network, the offset position of the touch position can be accurately predicted, and thus, based on the first touch position and the offset position, the touch position of the touch object on the touch screen can be accurately determined. Furthermore, the offset position prediction network is obtained based on the similarity between the offset positions of multiple frames of touch sensing image samples, thereby achieving the offset position prediction network through self-supervision in the absence of sub-pixel labeled data. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0019] Figure 1 This is a flowchart of a touch position prediction method provided by an embodiment of the present application; Figure 2 This is a touch sensing image provided by an embodiment of the present application when there is no touch; Figure 3 It is a touch sensing image provided by an embodiment of the present application when there is touch; Figure 4 is a flowchart of another touch position prediction method provided by an embodiment of the present application; Figure 5 This is an overall architecture diagram of a touch position prediction method provided by an embodiment of the present application; Figure 6 This is a training flow chart of an offset position prediction network provided in an embodiment of the present application; Figure 7 is a structural diagram of a touch device provided in an embodiment of the present application; Figure 8 is a structural diagram of a touch position prediction device provided in an embodiment of the present application; Figure 9 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0020] To make the above-mentioned purposes, features, and advantages of this application more clearly understood, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of this application.
[0021] The touch sensing image can be used to identify the stylus's pressed position. To ensure a smooth user interaction experience, the following two performance requirements must be met: 1) Fast response: The stylus's pressed state and precise position must be detected in real time at a relatively fast rate (e.g., 200 frames per second). 2) Sub-pixel accuracy: Because the resolution of screen display pixels (e.g., 800×1200 pixels) is much higher than that of the touch sensor, sub-pixel prediction accuracy is required. Specifically, the lower-resolution touch sensing image (e.g., 16×24 pixels) must be converted to the high-resolution pixel space of the display pixels through fine-grained mapping (e.g., 1 / 50) pixel accuracy to accurately locate the touch position. This mapping process requires not only speed but also extremely high precision to ensure that stylus operations produce natural and accurate feedback on the screen.
[0022] Existing touch position prediction schemes typically use simple computational methods (e.g., the centroid method) to achieve fast sub-pixel predictions. This method is computationally inexpensive but lacks sub-pixel accuracy. Machine learning significantly outperforms traditional hand-crafted algorithms in prediction performance. However, supervised learning for this task requires obtaining the sub-pixel touch position of the stylus as a learning target, which is difficult to obtain in real-world environments. The key technical approach to the touch position prediction method of the present application is to employ a self-supervised algorithm for this task, eliminating supervised learning's reliance on labeled data and achieving prediction performance that surpasses existing touch position prediction schemes.
[0023] The touch position prediction method according to the embodiment of the present application is described below with reference to the accompanying drawings.
[0024] Reference Figure 1 As shown, Figure 1 This is a flow chart of the steps of a touch position prediction method provided by an embodiment of the present application. Figure 1 As shown, a touch position prediction method provided by an embodiment of the present application may include steps S110 to S140: Step S110: Acquire a touch sensing image.
[0025] In the embodiments of the present application, the touch-sensing image can be a capacitive sensing image or a resistive sensing image. The embodiments of the present application do not limit the type of touch-sensing image. A touch-sensing image is generated by the touch-sensing element array below the touch screen based on a sensing signal after a touch object (e.g., a stylus) contacts the touch screen. The sensing signal can be capacitance or resistance. For example, if the sensing signal is capacitance, the capacitance of the touch-sensing element array below the touch screen changes after the touch object contacts the touch screen, thereby generating a capacitive sensing image (i.e., a touch-sensing image) based on the capacitance. For another example, if the sensing signal is resistance, the resistance of the touch-sensing element array below the touch screen changes after the touch object contacts the touch screen, thereby generating a resistive sensing image (i.e., a touch-sensing image) based on the resistance. To ensure a smooth user interaction experience, touch-sensing images are acquired at target time intervals; for example, touch-sensing images are acquired at intervals of 0.005 seconds.
[0026] For example, Figure 2 It illustrates a touch sensing image when there is no touch, wherein the fid in the touch sensing image indicates the acquisition order of the touch sensing image, and fid: 62 indicates that the touch sensing image is acquired for the 62nd time. Figure 3 It shows the touch sensing image when there is a touch. fid: 1306 means that the touch sensing image is acquired for the 1306th time. Figure 3 The pixels in the image are normalized to ensure uniform pixel values at non-touch locations.
[0027] Step S120: determining a first touch position of the touch object on the touch screen according to the touch sensing image.
[0028] In the embodiments of the present application, the first touch position is a position corresponding to a pixel of the touch sensing image and does not have sub-pixel accuracy. In some embodiments, the touch sensing image is processed according to a signal extreme value prediction method to obtain the first touch position of the touch object on the touch screen.
[0029] Step S130: predicting an offset position according to the target image area in the touch sensing image by using a pre-trained offset position prediction network, wherein the offset position prediction network learns the similarity between the offset positions of multiple frames of touch sensing image samples.
[0030] In an embodiment of the present application, the multi-frame touch sensing image samples refer to continuous multi-frame touch sensing image samples. Since the offset positions of the multi-frame touch sensing image samples are very similar, the offset position prediction network learns the similarity between the offset positions of the multi-frame touch sensing image samples. Therefore, the offset position prediction network learns a finer-grained (sub-pixel accuracy) offset position prediction and has better position prediction performance.
[0031] Furthermore, an offset position prediction network is obtained based on the similarity between the offset positions of multiple frames of touch sensing image samples, thereby achieving an offset position prediction network through self-supervision in the absence of sub-pixel labeled data.
[0032] The target image area refers to the image area including the first touch position. The offset position is obtained by inputting the target image area into a pre-trained offset position prediction network for offset position prediction; wherein the offset position is an offset position with sub-pixel accuracy.
[0033] Step S140: determining the touch position of the touch object on the touch screen according to the first touch position and the offset position.
[0034] In an embodiment of the present application, the first touch position is a position with the corresponding accuracy of a touch-sensing image pixel, and the offset position is an offset position with sub-pixel accuracy. By using the offset position to correct the first touch position, a touch position with sub-pixel accuracy can be obtained, and thus based on the touch position with sub-pixel accuracy, the touch position of the touch object on the touch screen can be determined; wherein, the touch position of the touch object on the touch screen is a position with sub-pixel accuracy.
[0035] In the above process, the first touch position of the touch object on the touch screen is determined based on the touch sensing image; and the offset position is predicted based on the target image area in the touch sensing image through a pre-trained offset position prediction network. Because the offset position prediction network learns the similarity between the offset positions of multiple frames of touch sensing image samples, the offset position prediction network has good position prediction performance. Based on the offset position prediction network, the offset position of the touch position can be accurately predicted, and thus, based on the first touch position and the offset position, the touch position of the touch object on the touch screen can be accurately determined. In addition, the offset position prediction network is obtained based on the similarity between the offset positions of multiple frames of touch sensing image samples, thereby achieving the offset position prediction network through self-supervision in the absence of sub-pixel labeled data.
[0036] The touch position prediction method of the embodiment of the present application is described below in Sections 3.1 and 3.2 respectively.
[0037] 3.1 Implementation of touch position prediction method: In conjunction with the above embodiments, in one embodiment, the present application further provides a touch position prediction method. In addition to the above steps, the method also includes the following steps: A target image area is extracted from the touch-sensing image according to the first touch position, where the target image area includes pixels corresponding to the first touch position in the touch-sensing image.
[0038] In an embodiment of the present application, the first touch position is a position with the corresponding accuracy of the touch sensing image pixels. There is an offset between the first touch position and the actual position. In order to obtain a more accurate touch position, the target image area is extracted from the touch sensing image according to the first touch position, and the offset position is predicted based on the target image area.
[0039] The pixel at the first touch position may be any pixel in the target image area. For example, the target image area is a 3×3 pixel area extracted from the touch-sensing image, and the pixel at the first touch position is any one of the 3×3 pixels. For another example, the target image area is a 3×4 pixel area extracted from the touch-sensing image, and the pixel at the first touch position is any one of the 3×4 pixels.
[0040] In some embodiments, extracting a target image area from the touch-sensing image according to the first touch position includes: extracting the target image area from the touch-sensing image with a pixel point corresponding to the first touch position in the touch-sensing image as the center.
[0041] For example, the target image area is an area consisting of 3×3 pixels extracted from the touch sensing image, wherein the pixel point of the first touch position is located in the middle of the 3×3 pixel area.
[0042] Through the above process, the target image area can be flexibly extracted from the touch sensing image based on the first touch position, so that the subsequent offset position prediction network can accurately predict the offset position based on the target image area without processing the entire touch sensing image. In this way, the computational complexity of the offset position prediction is reduced, and the introduction of interference information in other areas of the touch sensing image is avoided, thereby achieving more accurate offset position prediction.
[0043] In combination with the above embodiments, in one embodiment, the present application further provides a touch position prediction method. In this method, the resolution of the touch sensing image is the first resolution corresponding to the touch sensing element array of the touch screen; The step S120 of "determining the first touch position of the touch object on the touch screen according to the touch sensing image" may specifically include: determining the first touch position of the touch object on the touch screen according to the touch sensing image at the accuracy corresponding to the first resolution.
[0044] The first touch position is a position at a first resolution, which is a lower resolution than the touch screen resolution. In this case, the first touch position does not have sub-pixel accuracy. For example, if the touch screen's touch sensing element array corresponds to a first resolution of 16×24, the first touch position is a pixel position at the 16×24 resolution. The touch screen's display pixel resolution (e.g., 800×1200) is greater than the first resolution, making it difficult to determine the precise location of the touch on the touch screen based on the first touch position.
[0045] The step S130 of "predicting the offset position based on the target image area in the touch-sensing image through a pre-trained offset position prediction network" may specifically include: predicting the offset position based on the target image area through a pre-trained offset position prediction network at an accuracy corresponding to a second resolution, where the second resolution is the resolution of the display pixels of the touch screen, and the second resolution is greater than the first resolution.
[0046] In order to obtain a more accurate touch position, a pre-trained offset position prediction network is used to predict the offset position. Since the offset positions of multiple frames of touch sensing image samples are very similar, the offset position prediction network learns the similarity between the offset positions of multiple frames of touch sensing image samples. Therefore, the offset position prediction network learns a finer-grained (sub-pixel accuracy) offset position prediction and has better position prediction performance. Therefore, based on the offset position prediction network, the offset position can be predicted with an accuracy that meets the second resolution.
[0047] Through the above process, the precise touch position of the touch object on the touch screen is determined in two stages. First, the first touch position of the touch object on the touch screen is determined according to the touch sensing image at the accuracy corresponding to the first resolution of the touch sensing image. Then, according to the target image area, the offset position with sub-pixel accuracy is predicted through a pre-trained offset position prediction network at the accuracy corresponding to the second resolution of the touch screen. In this way, the first touch position is corrected by the offset position to obtain the touch position with sub-pixel accuracy, thereby determining the touch position of the touch object on the touch screen based on the touch position with sub-pixel accuracy.
[0048] In conjunction with the above embodiments, in one embodiment, the present application further provides a touch position prediction method. In this method, the first touch position includes a first coordinate position and a second coordinate position, and the offset position includes a first coordinate offset position and a second coordinate offset position. The above step S140 of "determining the touch position of the touch object on the touch screen based on the first touch position and the offset position" may specifically include: The first coordinate position and the first coordinate offset position are merged to obtain a first coordinate, and the second coordinate position and the second coordinate offset position are merged to obtain a second coordinate; and the touch position of the touch object on the touch screen is determined according to the first coordinate and the second coordinate.
[0049] In the embodiment of the present application, the touch position of the touch object on the touch screen is represented by two-dimensional coordinates, the first coordinate position is the position in the X-axis direction, and the second coordinate position is the position in the Y-axis direction; the offset position includes the offset position in the X-axis direction and the offset position in the X-axis direction.
[0050] Under the pixel correspondence accuracy of the touch sensing image, the first coordinate position and the second coordinate position are integer positions, and the first touch position can be expressed as (Xint, Yint); while the first coordinate offset position and the second coordinate offset position are sub-pixel offsets, and the offset position is expressed as (x0, y0). If the size of each pixel of the touch sensing image is 1, then -1.0 <x0<+1.0,-1.0<y0<+1.0。
[0051] The first coordinate position and the first coordinate offset position are fused (the first coordinate position and the first coordinate offset position are added) to obtain the first coordinate, and the second coordinate position and the second coordinate offset position are fused (the second coordinate position and the second coordinate offset position are added) to obtain the second coordinate, thereby determining the touch position of the touch object on the touch screen based on the first coordinate and the second coordinate.
[0052] The touch position prediction method of the embodiment of the present application is described below with reference to a specific embodiment. Figure 4 As shown, Figure 4 4 is a flowchart of another touch position prediction method provided by an embodiment of the present application, the method comprising the following steps S410 to S450: Step S410: Acquire a touch sensing image.
[0053] A touch-sensing image is generated by the touch-sensing element array beneath the touchscreen based on the sensing signal after a touch object (e.g., a stylus) contacts the touchscreen. To ensure a smooth user interaction experience, touch-sensing images are captured at target intervals; for example, they may be captured every 0.005 seconds.
[0054] Step S420: The resolution of the touch sensing image is a first resolution corresponding to the touch sensing element array of the touch screen. Under the accuracy corresponding to the first resolution, a first touch position of the touch object on the touch screen is determined according to the touch sensing image.
[0055] The first touch position is a position at a first resolution, which is a lower resolution than the resolution of the touch screen. In this case, the first touch position does not have sub-pixel accuracy. Specifically, the touch sensing image can be processed according to a signal extreme value prediction method to obtain the first touch position of the touch object on the touch screen.
[0056] Step S430: extracting a target image area from the touch-sensing image according to the first touch position, where the target image area includes pixels corresponding to the first touch position in the touch-sensing image.
[0057] In an embodiment of the present application, there is an offset between the first touch position and the actual position. In order to obtain a more accurate touch position, the target image area is extracted from the touch-sensing image to predict the offset position based on the target image area, wherein the pixel point of the first touch position can be any pixel point in the target image area.
[0058] Step S440: predicting an offset position according to the target image area through a predetermined offset position prediction network at an accuracy corresponding to a second resolution, where the second resolution is a resolution of display pixels of the touch screen, and the second resolution is greater than the first resolution.
[0059] In the embodiment of the present application, since the offset positions of multiple frames of touch-sensing image samples are very similar, the offset position prediction network learns the similarity between the offset positions of multiple frames of touch-sensing image samples, and thus the offset position prediction network learns a finer-grained (sub-pixel accuracy) offset position threshold, which has better position prediction performance. Therefore, based on the offset position prediction network, it is possible to predict the offset position with an accuracy that meets the second resolution.
[0060] Step S450: determining the touch position of the touch object on the touch screen according to the first touch position and the offset position.
[0061] Specifically, the first touch position includes a first coordinate position and a second coordinate position, and the offset position includes a first coordinate offset position and a second coordinate offset position. Thus, the first coordinate position and the first coordinate offset position are combined to obtain a first coordinate, and the second coordinate position and the second coordinate offset position are combined to obtain a second coordinate. The touch position of the touch object on the touch screen is determined based on the first coordinate and the second coordinate.
[0062] Since the offset position prediction network learns the similarity between the offset positions of multiple frames of touch sensing image samples, the offset position prediction network has better position prediction performance. Based on the offset position prediction network, the offset position of the touch position can be accurately predicted, so that the touch position of the touch object on the touch screen can be accurately determined based on the first touch position and the offset position.
[0063] Through the above process, the precise touch position of the touch object on the touch screen is determined in two stages. First, the first touch position of the touch object on the touch screen is determined based on the touch sensing image at the accuracy corresponding to the first resolution of the touch sensing image. Then, based on the target image area, the offset position is predicted with sub-pixel accuracy using a pre-trained offset position prediction network at the accuracy corresponding to the second resolution of the touch screen. Because the offset position prediction network learns the similarity between the offset positions of multiple frames of touch sensing image samples, the offset position prediction network has good position prediction performance. Based on the offset position prediction network, the offset position of the touch position can be accurately predicted, and thus, based on the first touch position and the offset position, the touch position of the touch object on the touch screen can be accurately determined.
[0064] For example, Figure 5 This is a diagram of the overall architecture of a touch position prediction method provided by an embodiment of the present application. First, the first touch position of the touch object on the touch screen is determined at the accuracy corresponding to the first resolution of the touch-sensing image. Next, the target image area is extracted based on the first touch position. Based on the accuracy corresponding to the second resolution of the touch screen, a pre-trained offset position prediction network is used to predict the offset position with sub-pixel accuracy based on the target image area. Finally, the first touch position and the offset position are fused to determine the touch position of the touch object on the touch screen.
[0065] 3.2 Training process of offset position prediction network: In combination with the above embodiments, in one embodiment, the present application also provides a touch position prediction method. In this method, in addition to the above steps, the following steps A1 to A2 are also included: Step A1: Acquire a training data set, where the training data set includes multiple frames of touch sensing image samples collected at target time intervals.
[0066] Step A2: using every two adjacent frames of touch sensing image samples in the training data set as a training sample, and performing self-supervisory training on the neural network to obtain the offset position prediction network.
[0067] In the embodiments of the present application, since the touch screen frame rate can reach very high levels (over 200 frames per second), it is reasonable to assume that the offset position predictions of touch-sensing image samples in adjacent frames are very close. Based on this assumption, there is no need to use actual sub-pixel annotated data. Instead, it is necessary to collect touch-sensing image samples from consecutive frames and provide the touch-sensing image samples from every two adjacent frames as input to a neural network. Based on the similarity of the offset position prediction results corresponding to the touch-sensing image samples in the two adjacent frames, the neural network parameters are optimized to obtain the offset position prediction network. In this way, a sub-pixel accurate offset position prediction network is obtained through self-supervised training in the absence of sub-pixel annotated data.
[0068] Specifically, every two adjacent frames of sensor image samples in the training data set are used as a training sample, and self-supervised training is performed on the neural network to obtain the offset position prediction network, including steps A21 to A22: Step A21: Input the target image area sample in the first frame touch-sensing image sample in the training sample into the neural network to obtain a first offset position prediction result, and input the target image area sample in the second frame touch-sensing image sample in the training sample into the neural network to obtain a second offset position prediction result, wherein the target image area sample includes pixel points corresponding to the sample touch position in the touch-sensing image sample.
[0069] Step A22: With the goal of maximizing the similarity between the first offset position prediction result and the second offset position prediction result, self-supervised training is performed on the neural network to obtain the offset position prediction network.
[0070] In the embodiment of the present application, after target image region samples are extracted from a first frame of touch-sensing image samples and a second frame of touch-sensing image samples, they are respectively input into a neural network for processing. The neural network outputs a first offset position prediction result and a second offset position prediction result. By calculating the difference between the first offset position prediction result and the second offset position prediction result (for example, the mean square error loss), the prediction results gradually converge and become consistent, thereby improving the accuracy of the sub-neural network and ultimately obtaining an offset position prediction network.
[0071] For example, Figure 6This is a training flowchart for an offset position prediction network provided in an embodiment of the present application. Target image region samples are extracted from the first and second touch-sensing image samples in the training samples. The target image region samples in the first touch-sensing image sample in the training samples are input into the neural network to obtain a first offset position prediction result. Furthermore, the target image region samples in the second touch-sensing image sample in the training samples are input into the neural network to obtain a second offset position prediction result. The offset position prediction network is obtained by calculating the mean squared error loss between the first and second offset position prediction results and performing self-supervised training on the neural network based on the mean squared error loss.
[0072] It is understandable that Figure 6 The two neural networks in refer to the same neural network. Figure 6 In order to illustrate that the first frame touch sensing image sample and the second frame touch sensing image sample in the training sample are processed twice, two neural networks are used for illustration.
[0073] In the above process, the network optimizes the similarity of the offset position prediction results of adjacent frames, which can achieve sub-pixel accuracy prediction without precise labeled data.
[0074] The present application also provides a touch device, referring to Figure 7 As shown, Figure 7 : is a structural diagram of a touch device provided in an embodiment of the present application, comprising: A touch screen comprising an array of touch sensing elements, wherein when a touch object contacts the touch screen, a sensing signal of the touch sensing element array changes, and a touch sensing image is generated according to the sensing signal; The processor is configured to execute the steps of the touch position prediction method described in the embodiment of the present application to determine the touch position of the touch object on the touch screen.
[0075] In an embodiment of the present application, a touch sensing element array is deployed below the touch screen. After a touch object contacts the touch screen, the sensing signal of the touch sensing element array changes, and a touch sensing image is generated according to the sensing signal. The processor determines the touch position of the touch object on the touch screen based on the touch sensing image.
[0076] The present application also provides a touch position prediction device, referring to Figure 8 As shown, Figure 8 : is a schematic structural diagram of a touch position prediction device provided in an embodiment of the present application, the device comprising: A first acquisition module 810 is configured to acquire a touch sensing image; A first determining module 820 is configured to determine a first touch position of the touch object on the touch screen according to the touch sensing image; a first prediction module 830 for predicting an offset position based on a target image region in the touch sensing image using a pre-trained offset position prediction network, wherein the offset position prediction network learns similarities between offset positions of multiple frames of touch sensing image samples; The second determining module 840 is configured to determine a touch position of the touch object on the touch screen according to the first touch position and the offset position.
[0077] In an optional embodiment, the device further includes: The first extraction module is configured to extract a target image area from the touch-sensing image according to the first touch position, where the target image area includes pixels corresponding to the first touch position in the touch-sensing image.
[0078] In an optional embodiment, the first extraction module is further configured to extract a target image area from the touch-sensing image with the pixel point corresponding to the first touch position in the touch-sensing image as the center.
[0079] In an optional embodiment, the device further includes: A second acquisition module is configured to acquire a training data set, wherein the training data set includes a plurality of touch sensing image samples collected at a target time interval; The training module is configured to use every two adjacent frames of touch sensing image samples in the training data set as a training sample, perform self-supervised training on the neural network, and obtain the offset position prediction network.
[0080] In an optional embodiment, the training module includes: a first input module, configured to input a target image area sample in a first frame of touch-sensing image samples in the training samples into the neural network to obtain a first offset position prediction result, and input a target image area sample in a second frame of touch-sensing image samples in the training samples into the neural network to obtain a second offset position prediction result, wherein the target image area sample includes pixel points corresponding to the sample touch position in the touch-sensing image sample; The training submodule is used to perform self-supervised training on the neural network with the goal of maximizing the similarity between the first offset position prediction result and the second offset position prediction result to obtain the offset position prediction network.
[0081] In an optional embodiment, the resolution of the touch sensing image is a first resolution corresponding to the touch sensing element array of the touch screen; The first determining module is further configured to determine a first touch position of the touch object on the touch screen according to the touch sensing image at an accuracy corresponding to the first resolution; The first prediction module is further used to predict the offset position according to the target image area through a pre-trained offset position prediction network with an accuracy corresponding to a second resolution. The second resolution is the resolution of the display pixels of the touch screen, and the second resolution is greater than the first resolution.
[0082] In an optional embodiment, the first touch position includes a first coordinate position and a second coordinate position, and the offset position includes a first coordinate offset position and a second coordinate offset position; and the second determining module includes: a fusion module, configured to fuse the first coordinate position and the first coordinate offset position to obtain a first coordinate, and to fuse the second coordinate position and the second coordinate offset position to obtain a second coordinate; The first confirmation submodule is configured to determine a touch position of the touch object on the touch screen according to the first coordinate and the second coordinate.
[0083] The present application also provides an electronic device, Figure 9 , Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 9 As shown, the electronic device 900 includes: a memory 910 and a processor 920. The memory 910 and the processor 920 are connected via a bus communication. A computer program is stored in the memory 910. The computer program can be run on the processor 920 to implement the steps of the touch position prediction method described in the embodiment of the present application.
[0084] The embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the touch position prediction method described in the embodiment of the present application are implemented.
[0085] The embodiment of the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the touch position prediction method described in the embodiment of the present application.
[0086] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0087] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods and devices according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0088] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.
[0089] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0090] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0091] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.
[0092] The above describes in detail a touch position prediction method, apparatus, touch device, electronic device, and storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only intended to help understand the method and core concept of the present application. At the same time, for those skilled in the art, based on the concept of the present application, there may be changes in the specific implementation methods and application scope. In summary, the contents of this specification should not be understood as limiting the present application.
Claims
1. A touch position prediction method, characterized in that: include: Acquire a touch sensing image; determining a first touch position of the touch object on the touch screen according to the touch sensing image; Predicting an offset position based on a target image region in the touch sensing image using a pre-trained offset position prediction network, wherein the offset position prediction network learns similarities between offset positions of multiple frames of touch sensing image samples; The touch position of the touch object on the touch screen is determined according to the first touch position and the offset position.
2. The method according to claim 1, characterized in that Also includes: A target image area is extracted from the touch-sensing image according to the first touch position, where the target image area includes pixels corresponding to the first touch position in the touch-sensing image.
3. The method according to claim 2, characterized in that Extracting a target image area from the touch-sensing image according to the first touch position includes: A target image area is extracted from the touch-sensing image with the pixel point corresponding to the first touch position in the touch-sensing image as the center.
4. The method according to claim 1, wherein Also includes: Acquire a training data set, wherein the training data set includes a plurality of touch sensing image samples collected at a target time interval; Every two adjacent frames of touch sensing image samples in the training data set are used as a training sample, and self-supervised training is performed on the neural network to obtain the offset position prediction network.
5. The method according to claim 4, characterized in that Using every two adjacent frames of touch sensing image samples in the training data set as a training sample, performing self-supervised training on the neural network to obtain the offset position prediction network, including: Inputting a target image area sample from a first frame of touch-sensing image samples in the training samples into the neural network to obtain a first offset position prediction result, and inputting a target image area sample from a second frame of touch-sensing image samples in the training samples into the neural network to obtain a second offset position prediction result, wherein the target image area sample includes pixel points corresponding to the sample touch position in the touch-sensing image sample; With the goal of maximizing the similarity between the first offset position prediction result and the second offset position prediction result, self-supervised training is performed on the neural network to obtain the offset position prediction network.
6. The method according to any one of claims 1 to 5, characterized in that: The resolution of the touch sensing image is a first resolution corresponding to the touch sensing element array of the touch screen; Determining a first touch position of the touch object on the touch screen according to the touch sensing image includes: determining, according to the touch-sensing image, a first touch position of the touch object on the touch screen at an accuracy corresponding to the first resolution; Predicting an offset position based on a target image area in the touch sensing image using a pre-trained offset position prediction network includes: At the accuracy corresponding to the second resolution, the offset position is predicted according to the target image area through a pre-trained offset position prediction network. The second resolution is the resolution of the display pixels of the touch screen, and the second resolution is greater than the first resolution.
7. The method according to any one of claims 1 to 5, characterized in that: The first touch position includes a first coordinate position and a second coordinate position, and the offset position includes a first coordinate offset position and a second coordinate offset position; Determining a touch position of the touch object on the touch screen according to the first touch position and the offset position includes: Merging the first coordinate position and the first coordinate offset position to obtain a first coordinate, and merging the second coordinate position and the second coordinate offset position to obtain a second coordinate; A touch position of the touch object on the touch screen is determined according to the first coordinate and the second coordinate.
8. A touch device, characterized in that: include: A touch screen comprising an array of touch sensing elements, wherein when a touch object contacts the touch screen, a sensing signal of the touch sensing element array changes, and a touch sensing image is generated according to the sensing signal; A processor is configured to execute the steps of the touch position prediction method according to any one of claims 1 to 7 to determine a touch position of a touch object on the touch screen.
9. A touch position prediction device, characterized in that: include: A first acquisition module, configured to acquire a touch sensing image; a first determining module, configured to determine a first touch position of the touch object on the touch screen according to the touch sensing image; a first prediction module, configured to predict an offset position based on a target image area in the touch sensing image using a pre-trained offset position prediction network, wherein the offset position prediction network learns similarities between offset positions of multiple frames of touch sensing image samples; The second determining module is configured to determine a touch position of the touch object on the touch screen according to the first touch position and the offset position.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the touch position prediction method according to any one of claims 1 to 7 are implemented.
11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the touch position prediction method according to any one of claims 1 to 7 are implemented.