Pressure input classification

Abstract

A method for classifying pressure inputs in a sensing array (201) is described, the sensing array including a plurality of sensing elements (202, 203, 204) responsive to pressure inputs. The method includes identifying a plurality of pressure inputs in the sensing array and converting the plurality of pressure inputs into an output image (601). The output image is compared by an artificial neural network to a dataset (701) including a plurality of images (702, 703, 704) of undesired pressure inputs. A mask (705) matching the output image is applied to remove the undesired pressure inputs.

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims priority to UK patent application no. 20 01 554.2, filed on 4 February 2020, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] The present invention relates to a method for classifying pressure inputs in a sensing array and to an apparatus for carrying out a method of this type. Force-sensing touchscreens are being used with increasing frequency in electronic devices such as mobile phones, tablet computers, and the like, in both personal and professional capacities. Typically, these types of electronic devices and touchscreens include a sensing array that includes a plurality of sensing elements. Summary of the Invention [Problem to be solved by the invention]

[0003] In situations where a user gestures with an input device, such as a stylus, on a sensing array, it is not uncommon for the user to apply pressure input to different portions of the sensing array that is additional to the pressure input generated from the pressure applied by the stylus. In particular, a user can generate pressure input from the stylus while simultaneously generating additional pressure input from the palm of their hand. Palm input is undesirable because it can lead to activations in the sensing array that do not correspond to the instructions the user wishes to provide. Specifically, the additional pressure input can introduce errors into the pressure calculation, distorting the desired input from the stylus and leaving the user with undesired or undesired output. [Means for solving the problem]

[0004] [Brief description of the invention] According to a first aspect of the present invention, a pressure sensor includes a plurality of sensing elements responsive to a pressure input. and executed by the processor.A method for classifying pressure inputs in a sensing array is provided, comprising: identifying a plurality of pressure inputs in the sensing array; converting the plurality of pressure inputs into an output image; comparing the output image by an artificial neural network to a dataset including a plurality of images including undesired pressure inputs, each of the images including an associated mask; and applying one of the associated masks that matches the output image in response to the comparing step to remove undesired pressure inputs; wherein converting the plurality of pressure inputs into an image comprises mapping each of the pressure inputs to a pixel of a pixel array corresponding to the sensing array.

[0005] According to a second aspect of the present invention, a sensing array including a plurality of sensing elements responsive to pressure inputs, a pixel array corresponding to the sensing array, and a processor, the processor configured to: identify a plurality of pressure inputs within the sensing array; convert the plurality of pressure inputs into an output image by mapping each of the pressure inputs to a pixel of the pixel array; compare the output image with a dataset including a plurality of images including undesired pressure inputs, each image including an associated mask, using an artificial neural network; and apply one of the associated masks corresponding to the output image in response to comparing the output images to remove the undesired pressure inputs. An apparatus for classifying pressure input is provided. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram illustrating an electronic device configured to classify a pressure input in accordance with the present invention. [Figure 2] FIG. 2 is a diagram of a sensing array of electronic devices that provides a response to an applied force or pressure. [Figure 3] FIG. 3 is a schematic exploded view of the sensing array of FIG. [Figure 4] FIG. 4 is a diagram illustrating the sensing array of FIG. 2 receiving pressure inputs, including desired and undesired inputs. [Figure 5] FIG. 5 is a diagram showing a pixel array corresponding to a sensing array. [Figure 6] FIG. 6 shows output images generated in response to multiple pressure inputs within the sensing array. [Figure 7] FIG. 7 is a schematic diagram illustrating the comparison of the output image with the dataset and the selection of an appropriate mask for application. [Figure 8] FIG. 8 illustrates how a mask can be applied to classify pressure inputs to avoid the influence of unwanted inputs. [Figure 9]FIG. 9 is a diagram illustrating a process by which the methods of the present invention can be incorporated into a method for tracking and visualizing inputs within a sensing array. DETAILED DESCRIPTION OF THE INVENTION

[0007] Embodiments of the present invention will now be described, by way of example only, with reference to the drawings. The detailed embodiments illustrate the best mode known to the inventors and support the claimed invention. However, they are merely exemplary and should not be used to interpret or limit the scope of the claims. Their purpose is to provide instruction to those skilled in the art. Elements and processes distinguished by ordinal phrases such as "first" or "second" do not necessarily define any order or ranking.

[0008] [Detailed Description of the Invention] (Figure 1) An apparatus for classifying pressure input according to the present invention can be utilized by an electronic device such as electronic device 101, shown in the form of a tablet computer. Electronic device 101 includes a touchscreen 102 that is responsive to applied force or pressure. Thus, a user 103 can use an input device in the form of a stylus 104 to provide an input pressure on touchscreen 102 and provide an appropriate output on touchscreen 102.

[0009] In use, the user 103 can provide various forms of pressure input from the stylus 104. This is often in the form of a gesture or swipe, and can include input such as a pattern in the form of a handwritten word constructed from successive gestures of the stylus 104 by the user 103. In further embodiments, the pressure input is provided in the form of a swipe, gesture and / or shape, or any other suitable form that can be produced by a suitable input device. It will be appreciated that input devices alternatives to a stylus can also be used in accordance with the present invention, such as a user's finger or a stylus that is not in the form of a pen as shown. When generating pressure input from the input device 104, the user 103 may have part of their hand resting on the electronic device 101 while writing or gesturing with the stylus 104, which may result in unwanted pressure input from the hand 105, particularly the palm of the hand. Thus, the electronic device 101, and consequently the touchscreen 102, may receive pressure input that includes not only the desired pressure input from the input device 104, but also unwanted input from the user 103.

[0010] (Figure 2) In the embodiment described with reference to FIG. 1, the electronic device 101 includes a sensing array 201 that responds to an applied force or pressure. Sensing array 201 includes a plurality of sensing elements, such as 202, 203, and 204. In an embodiment, each sensing element includes a pressure-sensitive material that responds to applied pressure. The pressure-sensitive material is of the type supplied by applicant Peratec Holdco Limited under the trademark QTC® and includes a material that exhibits a decrease in electrical resistance after the application of force or pressure. In this manner, the sensing array can be configured to provide both two-dimensional position data and range characteristics in response to applied pressure.

[0011] In this illustrated example, sensing array 201 includes 15 columns 205 and 5 rows 206. In a further exemplary embodiment, sensing array 201 includes 50 columns and 24 rows. It is further understood that alternative configurations are within the scope of the present invention and that any other suitable number of rows and columns may be utilized. Furthermore, while the illustrated example describes a square array, it should be understood that other alternative array formats may be utilized, such as a hexagonal array or the like. However, in all embodiments, the sensing array includes a plurality of sensing elements arranged to respond to the application of force or pressure. Column connectors 207 receive drive voltages from the processor, and row connectors 208 provide scan voltages to the processor. In the absence of applied force or pressure, all of the sense elements in sense array 201 remain non-conductive. However, when sufficient pressure is applied to the sense array adjacent to at least one of the sense elements, that sense element becomes conductive, thereby providing a response between the input drive line and the output scan line. In this manner, as user 103 moves input device 104 across touchscreen 102, and thereby the sense array 201, multiple sense elements may become conductive or activated in accordance with a hand gesture made with input device 104, as will be further described with respect to FIG. 4 .

[0012] (Figure 3) 3 shows a schematic exploded exemplary embodiment of the structure of sensing array 201. Sensing array 201 includes a first conductive layer 301 and a second conductive layer 302. Pressure-sensitive layer 303 is disposed between conductive layer 301 and conductive layer 302. In an embodiment, first conductive layer 301, second conductive layer 302, and pressure-sensitive layer 303 are sequentially printed as inks onto substrate 304 to form sensing array 301. First conductive layers 301 and 302 comprise a carbon-based material and / or a silver-based material, and the pressure-sensitive layer comprises a pressure-sensitive material, such as the type supplied by applicant Peratec Holdco Limited under the trademark QTC® shown above. Thus, the pressure-sensitive material may comprise a quantum tunneling composite material configured to exhibit a change in electrical resistance based on a change in applied force. The quantum tunneling composite material may be supplied as a printable ink or film.

[0013] The layers may be printed to form the pattern of the sensing array 201, as shown in plan view in Figure 2. In an embodiment, the first conductive layer 301 includes a plurality of conductive traces forming a plurality of rows that traverse the array in a first direction. In contrast, the second conductive layer 302 includes a further plurality of conductive traces forming a plurality of columns that traverse the array in a second direction. In an embodiment, the first and second directions are oriented at ninety degrees (90°) from each other. The pressure sensitive layer 303 is printed to provide a plurality of sensing elements formed at the intersections of the rows and columns of the first and second conductive layers. Thus, the sensing elements and pressure sensitive layer, in combination with the conductive layers, can provide a range characteristic or intensity of an applied force, such as a force in the direction of arrow 305, in a conventional manner by interpretation of the electrical output. Thus, when the user 103 applies a pressure input on the touch screen 102, the sensing array 201 provides an indicator of the magnitude of the applied force along with a position characteristic indicative of the location of the applied force. While Figures 1-3 illustrate exemplary electronic device and sensing array configurations suitable for the present invention, it is recognized that alternative electronic devices, touchscreens, and sensing arrays capable of providing both position and range characteristic outputs may also be used in accordance with the present invention.

[0014] (Figure 4) Figure 4 shows sensing array 201 receiving pressure input from user 103 in the manner of use shown in Figure 1. As highlighted in Figure 4, multiple sensing elements are activated in response to the pressure input provided by user 103. In the illustrated embodiment, sensing elements 401 and 402 are activated in response to pressure applied to the touchscreen 102 by the stylus 104, representing a desired input. Additionally, multiple additional sensing elements 403 in the sensing array 201 are activated, which are activated by pressure applied by the user 103 with the palm of their hand. This therefore represents an unwanted pressure input 404. As a result, when a user provides desired inputs 401 and 402 in combination with an unwanted pressure input, such as 404, the unwanted input may adversely affect calculations made regarding the desired input. For example, when applied forces are calculated by the sensing array 201 in the manner described in FIG. 3 , the unwanted pressure input may distort the desired input and output a pressure reading that does not match the pressure applied by the input device 104. As a result, a user may notice an inaccurate output or may discover that they have accidentally launched another program or application they did not intend. In an embodiment, multiple pressure inputs are identified on a frame-by-frame basis. Thus, once the sensing array 201 identifies both desirable and undesirable pressure inputs within the sensing array, the pressure inputs are converted into output images, which are further described with respect to FIGS.

[0015] (Figure 5) 5 shows a corresponding pixel array 501 that corresponds to the sensing array 201. The pixel array 501 includes a number of pixels that correspond to similar sensing elements in the sensing array 201. Thus, sensing element 401 corresponds to pixel 502, and sensing element 402 corresponds to pixel 503. In an embodiment, the pixels of pixel array 501 are arranged as a first plurality of pixels arranged in rows 504 and a second plurality of pixels arranged in columns 505. In this illustrated example, which corresponds to sensing array 201 above, pixel array 501 includes 15 columns and 5 rows. In a further exemplary embodiment, pixel array 501 includes 50 columns and 24 rows, consistent with similar embodiments of sensing arrays. It is further understood that alternative configurations are within the scope of the present invention and that other suitable numbers of rows and columns may be utilized. However, the arrangement is substantially similar to that of sensing array 201. Each pixel in pixel array 501 is configured to provide an output image. In an embodiment, the image may be output in a grayscale format, where the output is provided as either black or white. Thus, in this embodiment, pixels 502 and 503 are given an output corresponding to a desired input, and pixels 506 are given an output corresponding to an undesired input, as can be seen from the output image in FIG.

[0016] (Figure 6) 6 shows an output image 601 obtained from a pressure input to the sensing array 201. The output image 601 includes a desired input in the form of an image 602 and an undesired input in the form of an image 603. Thus, the output image 601 has been transformed from the multiple pressure inputs identified by the sensing array 201 by mapping each pressure input to a pixel in the pixel array 501. The output image 601 is provided in a grayscale format where pressure input is defined as black or white. In embodiments, a predetermined threshold determines the pressure input as either high or low. A high pressure input may be defined in accordance with a previously recorded pressure input that corresponds to the application of a palm or hand. A low pressure input may be defined in accordance with a previously recorded pressure input that corresponds to a desired stylus input, which is typically applied with a lower force than that resulting from a palm input. In embodiments, a high pressure input is defined as white with respect to the output image, and a low pressure input is defined as black with respect to the output image.

[0017] Thus, as can be seen from exemplary output image 601, activated sensing elements producing pressure input 404 are identified as being a high pressure input (above a predetermined palm threshold) and correspond to the resulting pressure colored white as undesired input 603. Similarly, activated sensing elements 401 and 402 are within a predetermined range of stylus pressure input and are therefore identified as being a low pressure input and are therefore output as desired input 602 colored black. This therefore ensures effective identification of unwanted inputs and application of appropriate masks for the unwanted inputs, as will be described with respect to FIG.

[0018] (Figure 7) The output image 601 can be processed by a processor to identify an appropriate mask to avoid the above-mentioned problems with unwanted pressure input. To accomplish this, the output image 601 is compared to a dataset 701. The dataset 701 includes a plurality of images containing unwanted pressure inputs. The dataset of images has been previously generated and provided to an artificial neural network (CANN). In one embodiment, the artificial neural network is a convolutional neural network (CNN). In an embodiment, an artificial neural network is trained based on the data set to identify white regions as undesired inputs and black regions as desired inputs, in this manner the artificial neural network is trained to identify undesired pressure inputs.

[0019] Consistent with traditional ANNs, this is accomplished by providing a large number of images in which desirable and undesirable pressure inputs are identified so that the ANN can distinguish between them. FIG. 7 shows a diagram of dataset 701 in reduced form. While only six such images are shown, in practice, the number of images provided may be much larger and may include hundreds or thousands of images of previously identified palm inputs. While generating corresponding output images from actual inputs may be time-consuming, dataset 701 can be expanded by providing additional images incrementally adjusted from the actual input images. For example, dataset 701 shows output image 702 and output image 703. Output image 703 is the inverted version of output image 702. This provides the ANN with two images in the dataset from a single actual input. Thus, in this manner, images can be adjusted to augment the dataset. While this illustrated example identifies image adjustments resulting from flipping an image, it is understood that a dataset can be augmented by rotating, scaling, or otherwise modifying the actual image.

[0020] Thus, the artificial neural network compares output image 601 with dataset 701 and identifies image 704 as similar to output image 601. Each image in dataset 701 has an associated mask corresponding to it. In this manner, once the artificial neural network has identified a corresponding image, its associated mask can be identified and applied to output image 601. The mask associated with each image in the dataset 701 is generated by identifying undesired pressure images in the dataset and providing them in a grayscale format similar to black and white. The mask can be expanded as needed by adjusting the kernel size of the artificial neural network to increase the boundaries of areas detected as undesired. This allows unwanted pressure inputs outside the boundaries of the mask to be avoided during removal.

[0021] For each image in the dataset, an appropriate mask is generated, which in an embodiment is a single-channel additional image with a bit depth of 1 bit (black or white). Each mask is generated manually by an operator using undesired inputs identified and highlighted as 1 or 0, corresponding to black or white, respectively. Mask 705 represents a mask associated with image 704. After comparing output image 601 to dataset 701 and identifying similarities with image 704, mask 705 can be applied to remove undesired pressure inputs corresponding to undesired pressure inputs 603 in output image 601. In this way, pressure inputs received below the mask can be removed from the output from the sensing array, thereby highlighting readings from desired inputs without erroneous readings, which can be more accurately interpreted.

[0022] (Figure 8) 8 illustrates a schematic of a method for classifying pressure inputs in a sensing array as described herein. In step 801, a plurality of pressure inputs are identified, including both desired and undesired pressure inputs. In step 802, the multiple pressure inputs are converted into appropriate output images, such as output image 601 described above. Once the output image is generated, it is compared to a dataset, such as dataset 701. The dataset includes multiple images that contain undesirable pressure inputs, and in step 803, an artificial neural network compares the previously identified output images to the dataset.

[0023] Each dataset image contains an associated mask, which can be applied to remove any unwanted pressure inputs that were generated. By adjusting the kernel size within the artificial neural network, we can ensure that all unwanted inputs are effectively removed from the output so that the desired inputs are not obstructed when providing the output to the user. It should be understood that the classification process described herein is by way of example, and that alternative sensing arrangements, touchscreens, and electronics can be utilized to provide similar pressure inputs and outputs in accordance with the present invention. Additionally, while the undesired input is described herein as originating from the palm of a user's hand, it should be further understood that in other applications, the undesired input may be generated by other means, such as other inputs that do not require activating the touchscreen in the required manner. Accordingly, it should be understood that alternative, functionally similar embodiments are within the scope of this application and the present invention.

[0024] (Figure 9) The methods for classifying pressure inputs described above can be incorporated as part of a process that can be utilized for multi-input force sensing. Figure 9 illustrates a process in which the methods of the claimed invention can be incorporated into a method for tracking and visualizing inputs within a sensing array. In step 901, a sensing array, such as sensing array 201, is initialized. Once the necessary algorithms to ensure smooth, predictable output to the electronic device or touchscreen are loaded, the touchscreen is ready to respond to any touch by the user. Characteristics of the sensing array may be provided to the processor to ensure that any output is compatible with the sensing array itself.

[0025] According to the invention described herein, a palm rejection model can also be loaded to effectively reject readings made by the user's palm in error and not mistaken for genuine input. In step 902, data is collected from the sensing array to ascertain background levels that need to be filtered from the input. The data collection process extracts raw data from the sensor, including stray activations, sensor noise, and the desired signal. Before determining the location of a genuine touch input, any stray activations and / or sensor noise, including palm readings, must be filtered out to avoid readings prone to significant noise.

[0026] During this step, in one embodiment, a low-pass convolution filter is applied to the collected data to smooth out any noisy background. For initial acquisition or when there is currently no input from the user's finger or stylus, background readings can be saved to update previous background readings previously saved in the system. The background readings can then be removed from the current data, along with any data that exceeds a threshold of unwanted data input, such as a palm, as discussed above, per step 903. In step 904, the coordinates of any non-zero input are defined. This provides an isolated signal free from the background data and undesired input previously removed in step 903. The data identified here is considered to be desired input that forms part of the user's intended input path. In step 905, the type of input is identified. This can be either a new pressure input, a current pressure input (e.g., a continuation of a pressure input path), a predicted input (based on previous input data), or no input.

[0027] Once the pressure input has been identified, smoothing can be applied in step 906. The current data point is weighted to provide an output that is stable against noise in the system. The smoothing applied does not distort the output if the previously identified undesired input has been removed from the output, providing a clean, desirable output reading. In step 907, the pressure input path is recorded by the system, and in step 908, an output visualization is provided to the user. Prior to output, to provide a better visualization to the user, Bezier smoothing can be applied to increase the apparent resolution and improve the appearance of the output to the user. In instances where the output represents the user's handwriting, generated by, for example, the stylus 104, the appearance provides a more natural appearance to the handwritten image. Bezier smoothing is particularly advantageous because it can include a continuous approach that checks and updates historical data while the pressure input path is still complete. This allows for fine adjustments that are particularly useful when outputting handwritten pressure inputs. Once the output has been provided, the process will repeat, returning to step 902, to provide a continuing output to the user.

Claims

1. 1. A method of classifying pressure inputs in a sensing array including a plurality of sensing elements responsive to the pressure inputs, the method being performed by a processor, the method comprising: identifying a plurality of pressure inputs within the sensing array; converting the plurality of pressure inputs into an output image; comparing the output image with a dataset comprising a plurality of images comprising undesired pressure inputs by an artificial neural network, wherein each of the images comprises an associated mask; and applying one of the associated masks corresponding to the output image in response to the comparing step to remove unwanted pressure inputs; converting the plurality of pressure inputs into an image includes mapping each of the pressure inputs to a pixel of a pixel array corresponding to the sensing array. A method comprising:

2. The method of claim 1 further comprising generating the image in grayscale format.

3. 3. The method of claim 2, wherein the pressure input is defined as high or low, and further wherein a high pressure input is designated as white and a low pressure input is designated as black.

4. The method of claim 1 , further comprising generating the data set including a plurality of images including an undesired pressure input.

5. The method of claim 4 , wherein the step of generating the data set comprises augmenting the data set by adjusting each of the images.

6. The method of claim 5 , wherein adjusting each image comprises at least one of: rotating the image; flipping the image; and scaling the image.

7. 7. The method of claim 4, further comprising training the artificial neural network through the data set to identify undesirable pressure inputs.

8. 8. The method of claim 1, further comprising the step of expanding the mask by adjusting a kernel size of the artificial neural network.

9. The method of claim 1 , further comprising generating the mask by identifying undesired pressure inputs from the plurality of images in the data set.

10. The method of claim 1 , wherein the step of identifying the plurality of pressure inputs occurs on a frame-by-frame basis.

11. 11. The method of claim 1, wherein the artificial neural network is a convolutional neural network.

12. a sensing array including a plurality of sensing elements responsive to a pressure input, a pixel array corresponding to the sensing array, and a processor; the processor: identifying a plurality of pressure inputs within the sensing array; converting the plurality of pressure inputs into an output image by mapping each pressure input to a pixel of the pixel array; comparing the output image with a dataset comprising a plurality of images comprising undesired pressure inputs by an artificial neural network, wherein each image comprises an associated mask; and applying one of the associated masks corresponding to the output image in response to comparing the output image to remove unwanted pressure inputs. It is configured as follows: A device that classifies pressure input.

13. The device of claim 12 , wherein the sensing array is configured to provide two-dimensional position data and range characteristics in response to applied pressure.

14. 14. Apparatus according to claim 12 or claim 13, wherein the mask has a bit depth of 1 bit.

15. 15. Apparatus according to any one of claims 12 to 14, wherein the image is generated in greyscale format.

16. A touchscreen comprising a device according to any one of claims 12 to 15.

17. 17. An electronic device comprising the touchscreen of claim 16.

Images