Pen state detection circuit and method, and input system

Abstract

To provide a pen state detection circuit and method, and an input system that can improve accuracy of estimating a pen state of an electronic pen having at least one electrode.SOLUTION: A pen state detection circuit (36) executes: a combination step (SP14) of combining a first signal distribution (90) and a second signal distribution (92) to generate a two-dimensional signal distribution (94), or combining a first feature quantity distribution related to a shape of the first signal distribution (90) and a second feature quantity distribution related to a shape of the second signal distribution (92) to generate a two-dimensional feature quantity distribution; and an estimation step (SP16, 18, 20) of estimating an indication position or attitude of an electronic pen (14) from the two-dimensional signal distribution (94) or the two-dimensional feature quantity distribution obtained through the combination by using predetermined input / output models (64A-64D).SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a pen state detection circuit and method, as well as an input system. [Background technology]

[0002] Conventional writing input systems combining an electronic pen and electronic equipment have been known. In this type of system, it is desirable that the position indicated by the electronic pen be accurately detected by the electronic equipment. For example, Patent Document 1 discloses electronic equipment that detects a first position where the user's hand makes contact and a second position indicated by the electronic pen on the detection surface of a touch sensor, estimates the tilt direction of the electronic pen using the coordinate values ​​of the first and second positions, and corrects the position indicated by the electronic pen according to this tilt direction. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2015-087785 [Overview of the project] [Problems that the invention aims to solve]

[0004] For example, by using an electronic pen with two electrodes, it is possible to estimate the position and orientation of the electronic pen even when the user's hand is not touching the detection surface. However, since the two electrodes are physically separated, at least one electrode will not always be in contact with the detection surface while the electronic pen is in use. In this case, the relationship between the tilt angle of the electronic pen and the detection position may change depending on the three-dimensional shape of the electrodes, so the estimation accuracy may vary depending on the position and orientation of the electronic pen.

[0005] The present invention has been made in view of these problems, and its object is to provide a pen state detection circuit and method, as well as an input system, that can improve the accuracy of estimating the pen state in an electronic pen having at least one electrode. [Means for solving the problem]

[0006] A pen state detection circuit in a first aspect of the present invention is a circuit that detects the state of an electronic pen having at least one electrode based on a signal distribution detected by a capacitive touch sensor having a plurality of sensor electrodes arranged in a planar manner, and performs an acquisition step of acquiring a one-dimensional signal distribution showing a change in capacitance as the electrode approaches, the first signal distribution along a first direction on the touch sensor, and the second signal distribution along a second direction intersecting the first direction from the touch sensor; a synthesis step of generating a two-dimensional signal distribution by combining the acquired first signal distribution and the second signal distribution, or generating a two-dimensional feature distribution by combining a first feature distribution relating to the shape of the first signal distribution and a second feature distribution relating to the shape of the second signal distribution; and an estimation step of estimating the instruction position or orientation of the electronic pen from the synthesized two-dimensional signal distribution or the two-dimensional feature distribution using a predetermined input / output model.

[0007] Furthermore, in the synthesis step, the two-dimensional signal distribution may be generated by multiplying a first corresponding value corresponding to the position in the first direction in the first signal distribution by a second corresponding value corresponding to the position in the second direction in the second signal distribution, or the two-dimensional feature distribution may be generated by multiplying a first corresponding value corresponding to the position in the first direction in the first feature distribution by a second corresponding value corresponding to the position in the second direction in the second feature distribution.

[0008] Furthermore, the input / output model may also be a machine learning model that includes a convolutional neural network that takes the two-dimensional signal distribution or the two-dimensional feature distribution as input and outputs coordinate values ​​indicating the position indicated by the electronic pen.

[0009] Furthermore, the input / output model may also be a machine learning model that includes a convolutional neural network that takes the two-dimensional signal distribution or the two-dimensional feature distribution as input and outputs at least one of the tilt angle and rotation angle that indicate the orientation of the electronic pen.

[0010] Furthermore, the input / output model may also be a machine learning model that includes: a convolutional neural network that takes the two-dimensional signal distribution or the two-dimensional feature distribution as input and outputs at least one of the tilt angle and rotation angle indicating the orientation of the electronic pen; a first-layer neural network that takes at least one of the tilt angle and rotation angle output from the convolutional neural network and the first signal distribution or the first feature distribution as input and outputs coordinate values ​​in the first direction indicating the indicated position of the electronic pen; and a second-layer neural network that takes at least one of the tilt angle and rotation angle output from the convolutional neural network and the second signal distribution or the second feature distribution as input and outputs coordinate values ​​in the second direction indicating the indicated position of the electronic pen.

[0011] Furthermore, the electronic pen may be provided with a plurality of electrodes, and the two-dimensional signal distribution or the two-dimensional feature distribution may be composed of an image signal having a plurality of color channels corresponding to the electrodes.

[0012] Furthermore, the first electrode may be a tip electrode having a shape symmetrical to the shaft of the electronic pen and provided at the tip of the electronic pen, and the second electrode may be an upper electrode having a shape symmetrical to the shaft of the electronic pen and provided on the proximal end side of the tip electrode.

[0013] The pen state detection method according to the second aspect of the present invention is a method for detecting the state of an electronic pen having at least one electrode based on a signal distribution detected by a capacitance-type touch sensor in which a plurality of sensor electrodes are arranged in a planar shape, the method comprising: an acquisition step of acquiring from the touch sensor a one-dimensional signal distribution indicating a change in capacitance accompanying the approach of the electrode, a first signal distribution along a first direction on the touch sensor, and a second signal distribution along a second direction intersecting the first direction; a synthesis step of generating a two-dimensional signal distribution by synthesizing the acquired first signal distribution and second signal distribution, or generating a two-dimensional feature amount distribution by synthesizing a first feature amount distribution regarding the shape of the first signal distribution and a second feature amount distribution regarding the shape of the second signal distribution; and an estimation step of estimating the indication position or posture of the electronic pen from the synthesized two-dimensional signal distribution or two-dimensional feature amount distribution using a predetermined input-output model, which is executed by one or a plurality of processors.

[0014] The input system according to the third aspect of the present invention includes an electronic device configured to include the above-described pen state detection circuit, an electronic pen used together with the electronic device, and a server device capable of storing model parameters for describing an input-output model constructed in the pen state detection circuit and communicable bidirectionally with the electronic device. When the electronic pen is detected, the electronic device requests the server device to transmit model parameters corresponding to the electronic pen.

Advantages of the Invention

[0015] According to the present invention, it is possible to improve the estimation accuracy of the pen state in an electronic pen having at least one electrode.

Brief Description of the Drawings

[0016] [Figure 1] It is an overall configuration diagram of an input system in an embodiment of the present invention. [Figure 2] It is a sequence diagram regarding the operation of the input system shown in FIG. 1. [Figure 3]It is a block diagram showing an example of the configuration of an electronic device in FIG. 1. [Figure 4] It is a functional block diagram of the detection unit shown in FIG. 3. [Figure 5] It is a flowchart regarding the detection operation of the pen state by the touch IC. [Figure 6] It is a schematic diagram partially showing the electronic pen in FIGS. 1 and 3. [Figure 7] It is a diagram showing an example of the 1D signal distribution acquired from the touch sensor. [Figure 8] It is a diagram showing an example of the local distribution detected from the touch sensor in the contact state of the electronic pen. [Figure 9] It is a diagram showing an example of the local distribution detected from the touch sensor in the contact state of the electronic pen. [Figure 10] It is a diagram showing an example of a method for synthesizing the 1D signal distribution. [Figure 11] It is a diagram showing an example of a method for calculating the 1D feature amount distribution. [Figure 12] It is a diagram showing the network structure constituting the input / output model of the first example. [Figure 13] It is a diagram showing the network structure constituting the input / output model of the second example. [Figure 14] It is a diagram showing the network structure constituting the input / output model of the third example. [Figure 15] It is a diagram showing the network structure constituting the input / output model of the fourth example. [Mode for Carrying Out the Invention]

[0017] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. For ease of understanding of the description, the same reference numerals are attached to the same components in each drawing as much as possible, and duplicate descriptions are omitted.

[0018] [[ID=]47] [Description of Input System 10] [Overall Configuration] Figure 1 is an overall configuration diagram of an input system 10 in one embodiment of the present invention. This input system 10 is configured to receive writing force using an electronic device 12 and an electronic pen 14 and to generate digital ink (or ink data) with high reproducibility of writing input. Specifically, this input system 10 comprises at least one electronic device 12, at least one electronic pen 14, and a server device 16. Each electronic device 12 can communicate with the server device 16 via a network NT.

[0019] The electronic device 12 is a general-purpose or dedicated device equipped with a touch panel display 32 (Figure 3). Examples of general-purpose devices include tablet terminals, smartphones, and personal computers. On the other hand, examples of dedicated devices include digital signage (so-called electronic billboards) and wearable devices.

[0020] The electronic pen 14 is a pen-shaped pointing device configured to communicate unidirectionally or bidirectionally via electrostatic coupling formed with the electronic device 12. The user can write pictures or characters on the electronic device 12 by holding the electronic pen 14 and moving it while pressing the pen tip against a predetermined touch surface. This electronic pen 14 is, for example, an active electrostatic coupling (AES) or electromagnetic induction (EMR) stylus.

[0021] The server device 16 is a computer that performs overall control related to the provision of model parameters 20, and may be either cloud-based or on-premise. Here, the server device 16 is illustrated as a single computer, but instead, the server device 16 may be a group of computers that make up a distributed system. Specifically, the server device 16 is composed of a server-side communication unit 22, a server-side control unit 24, and a server-side storage unit 26.

[0022] The server-side communication unit 22 is an interface for sending and receiving electrical signals to and from external devices. This allows the server device 16 to receive model selection information 18 from the electronic device 12 and transmit model parameters 20 to the electronic device 12.

[0023] The server-side control unit 24 is composed of a processing unit including a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The server-side control unit 24 functions as a model selection unit 28, which will be described in detail later, by reading and executing a program stored in the server-side storage unit 26.

[0024] The server-side storage unit 26 consists of a non-transient and computer-readable storage medium, such as a hard disk drive (HDD) or a solid-state drive (SSD). As a result, a database (hereinafter referred to as the parameter DB 30) relating to the model parameters 20 is constructed in the server-side storage unit 26.

[0025] <Basic operation> The input system 10 in this embodiment is configured as described above. Next, the operation of the input system 10, specifically the coordinated operation of the electronic device 12, the electronic pen 14, and the server device 16, will be explained with reference to the sequence diagram in Figure 2. Steps S1 and S9 in this sequence diagram are performed by the cooperation of the electronic pen 14 and the electronic device 12, while steps S4 to S6 are performed by the server device 16. The remaining steps are performed by the electronic device 12.

[0026] In step S1 of Figure 2, the electronic device 12 detects the electronic pen 14 used for the user's writing input. Specifically, the electronic device 12 attempts to pair with the electronic pen 14 located in its vicinity and detects the electronic pen 14 through the success of this pairing. Alternatively, the electronic device 12 may detect the electronic pen 14 by accepting an operation from the user to input information about the electronic pen 14.

[0027] In step S2, the electronic device 12 obtains model selection information 18 from the electronic pen 14 and / or the electronic device 12 itself.

[0028] In step S3, the electronic device 12 transmits data including the model selection information 18 acquired in step S2 to the server device 16, in association with the identification information of the electronic device 12 (i.e., the device ID).

[0029] In step S4, the server device 16 acquires model selection information 18 by receiving data from the electronic device 12.

[0030] In step S5, the server-side control unit 24 (more specifically, the model selection unit 28) uses the model selection information 18 obtained in step S4 as a search key to refer to the parameter DB 30 constructed in the server-side storage unit 26. As a result, one set of model parameters 20 is selected from among multiple sets of model parameters 20 that can identify the input / output model corresponding to the model selection information 18.

[0031] In step S6, the server device 16 transmits data including the model parameters 20 selected in step S5 to the electronic device 12 having a device ID associated with the corresponding model selection information 18.

[0032] In step S7, the electronic device 12 obtains model parameters 20 through the reception of data from the server device 16. These model parameters 20 are stored in the memory 42 (Figure 3) of the electronic device 12.

[0033] In step S8, the electronic device 12 sets the model parameters 20 obtained in step S7 in a manner that is usable by the electronic device 12 itself. For example, the host processor 40 (Figure 3) writes each value of the model parameters 20 to the corresponding memory or register storage area.

[0034] In step S9, the electronic device 12 performs the desired writing operation in cooperation with the electronic pen 14. As a result, the electronic device 12 can perform the generation process of digital ink 56 (Figure 3) and the drawing process using the digital ink 56, with the model parameters 20 corresponding to itself set.

[0035] [Electronic Device 12 Description] <Block diagram> Figure 3 is a block diagram showing an example of the configuration of the electronic device 12 in Figure 1. Specifically, this electronic device 12 includes a touch panel display 32, a display driver IC (Integrated Circuit) 34, a touch IC 36 (corresponding to the "pen state detection circuit"), a communication module 38, a host processor 40, and a memory 42.

[0036] The touch panel display 32 comprises a display panel 44 capable of visibly displaying content, and a planar touch sensor 46 positioned to overlap the display panel 44 in a planar view. In the example shown in this figure, the touch sensor 46 is an "external" sensor attached to the display panel 44 from the outside, but it may be replaced with an "internal" sensor (further classified as on-cell or in-cell) that is integrated with the display panel 44.

[0037] The display panel 44 is capable of displaying monochrome or color images and is composed of, for example, a liquid crystal panel, an organic EL (Electro-Luminescence) panel, or electronic paper. By making the display panel 44 flexible, the user can perform handwriting input operations while the touch surface of the electronic device 12 is curved or bent.

[0038] The touch sensor 46 is a capacitive sensor comprising a plurality of sensor electrodes 46s (Figure 6) arranged in a planar manner. This touch sensor 46 is configured to include, for example, a plurality of X-line electrodes for detecting the position in the X direction (first direction) of the sensor coordinate system, and a plurality of Y-line electrodes for detecting the position in the Y direction (second direction) intersecting the X direction. In this case, the X-line electrodes are provided extending in the Y direction and are arranged at equal intervals along the X direction. The Y-line electrodes are provided extending in the X direction and are arranged at equal intervals along the Y direction. Note that the touch sensor 46 may be a self-capacitive sensor in which block-shaped electrodes are arranged in a two-dimensional grid, instead of the mutual-capacitive sensor described above.

[0039] The display driver IC 34 is an integrated circuit that is electrically connected to the display panel 44 and controls the driving of the display panel 44. The display driver IC 34 drives the display panel 44 based on display signals supplied from the host processor 40. As a result, the content indicated by the digital ink 56 is displayed on the display panel 44.

[0040] The touch IC 36 is an integrated circuit that is electrically connected to the touch sensor 46 and controls the operation of the touch sensor 46. The touch IC 36 drives the touch sensor 46 based on control signals supplied from the host processor 40. As a result, the touch IC 36 functions as a detection unit 50 that performs "pen detection" to detect the state of the electronic pen 14 and "touch detection" to detect touches made by the user's finger or the like.

[0041] The "pen detection" function includes, for example, a scanning function for the touch sensor 46, a function for receiving and analyzing downlink signals, a function for estimating the state of the electronic pen 14 (e.g., position, orientation, pressure), and a function for generating and transmitting uplink signals containing commands for the electronic pen 14. The "touch detection" function also includes, for example, a two-dimensional scanning function for the touch sensor 46, a function for acquiring a detection map on the touch sensor 46, and a function for classifying areas on the detection map (e.g., classification of fingers, palms, etc.).

[0042] In this way, a graphical user interface (GUI) is constructed by combining the input functions of the electronic pen 14 and touch sensor 46 with the output functions of the display panel 44.

[0043] The communication module 38 has a communication function that enables wired or wireless communication with external devices. As a result, the electronic device 12 can transmit model selection information 18 to the server device 16 and receive model parameters 20 from the server device 16.

[0044] The host processor 40 is composed of a processing unit including a CPU, GPU, or MPU (Micro-Processing Unit). The host processor 40 functions as an ink generation unit 52 and a drawing processing unit 54 by reading and executing an information processing program stored in memory 42. The ink generation unit 52 performs generation processing to generate digital ink 56 based on detection data sequentially supplied from touch IC 36. The drawing processing unit 54 performs drawing processing to draw content within the display area of ​​the display panel 44 using the time series of detection data.

[0045] The memory 42 is non-transient and consists of a computer-readable recording medium (or storage medium). Here, the computer-readable recording medium is a storage device including an HDD, or a portable medium such as a magneto-optical disk, ROM (Read Only Memory), CD (Compact Disc)-ROM, or flash memory. In the example shown in this figure, the memory 42 stores model selection information 18, model parameters 20, and digital ink 56.

[0046] The model selection information 18 is information necessary for selecting the input / output model to be built on the touch IC 36, and specifically includes [1] identification information relating to at least one of the types of electronic pen 14, electronic device 12, and touch sensor 46, and the user, and [2] specification information relating to the specifications of the electronic pen 14 or electronic device 12.

[0047] Model parameters 20 are a set of parameters that describe the input / output model built on the touch IC 36. Each value of model parameters 20 is predetermined through machine learning. Model parameters 20 consist of "variable parameters," such as coefficients describing the activation functions of the arithmetic units and the coupling strength between arithmetic units, and "fixed parameters" (so-called hyperparameters) that identify the architecture of the learning model. Examples of hyperparameters include the number of arithmetic units that make up each layer and the number of hidden layers. For example, if the architecture is fixed, model parameters 20 may consist only of variable parameters.

[0048] Digital Ink 56 is ink data for representing handwritten content. Examples of "ink description languages" used to describe Digital Ink 56 include WILL (Wacom Ink Layer Language), InkML (Ink Markup Language), and ISF (Ink Serialized Format).

[0049] <Functional block of detection unit 50> Figure 4 is a functional block diagram of the detection unit 50 shown in Figure 3. This detection unit 50 is composed of a distribution acquisition unit 60, a distribution synthesis unit 62, and an estimator 64.

[0050] The distribution acquisition unit 60 acquires a one-dimensional signal distribution (hereinafter also referred to as "1D signal distribution") from the touch sensor 46 that shows the change in capacitance (more specifically, mutual capacitance or self-capacitance) as the electrodes of the electronic pen 14 approach. The 1D signal distribution corresponds to the cross-sectional shape of the two-dimensional map detected on the touch sensor 46. Hereinafter, in order to distinguish between the types of 1D signal distributions, the signal distribution along the X direction on the touch sensor 46 may be called the "X signal distribution," and the signal distribution along the Y direction on the touch sensor 46 may be called the "Y signal distribution."

[0051] The distribution synthesis unit 62 generates a two-dimensional signal distribution (hereinafter also referred to as "2D signal distribution") by synthesizing the X signal distribution and Y signal distribution acquired by the distribution acquisition unit 60. Specifically, the distribution synthesis unit 62 [1] performs interpolation on the X signal distribution to obtain the signal value corresponding to the X coordinate of the calculation target (hereinafter also referred to as "first corresponding value"), [2] performs interpolation on the Y signal distribution to obtain the signal value corresponding to the Y coordinate of the calculation target (hereinafter also referred to as "second corresponding value"), and [3] synthesizes the first corresponding value and the second corresponding value (specifically, by multiplication or addition).

[0052] The distribution synthesis unit 62 may, if necessary, calculate feature quantities relating to the shape of the one-dimensional signal distribution (hereinafter also referred to as "one-dimensional feature distribution" or "1D feature distribution") and synthesize these 1D feature distributions together. In this case, a two-dimensional feature distribution (hereinafter also referred to as "2D feature distribution") is generated by synthesizing the "X feature distribution" corresponding to the X signal distribution and the "Y feature distribution" corresponding to the Y signal distribution. The synthesis of feature distributions is performed in the same manner as in the case of the signal distribution described above. Examples of features include various features that characterize the shape of the signal distribution, such as the slope of the signal distribution or the absolute value of the slope.

[0053] The estimator 64 estimates the state of the electronic pen 14 (hereinafter also referred to as the "pen state") from the 2D signal distribution or 2D feature distribution synthesized by the distribution synthesis unit 62. This pen state includes [1] the indicated position of the electronic pen 14 (e.g., coordinate values ​​on the sensor coordinate system), [2] the orientation of the electronic pen 14 (e.g., angle and direction), and [3] the amount of change over time of the indicated position and orientation (e.g., velocity and acceleration). The estimator 64 is composed of a learner whose calculation rules are defined by the values ​​of the model parameters 20 (Figure 1). Specifically, the estimator 64 functions as a feature map generation unit 66, a data output unit 68, and a data interpretation unit 70.

[0054] The feature map generation unit 66 generates a feature map from the synthesized 2D signal distribution or 2D feature distribution, showing features related to the shape of the distribution. This feature map is composed of voxel data of an arbitrary size (X number of pixels / Y number of pixels / number of channels).

[0055] The data output unit 68 outputs data (hereinafter referred to as "output data") that shows the result of estimating the pointing position or orientation of the electronic pen 14 from the feature map generated by the feature map generation unit 66. This output data may be defined as an absolute value (e.g., 100 mm) indicating the pointing position or orientation of the electronic pen 14, or as a relative value (e.g., a value normalized in the range [0,1]).

[0056] The data interpretation unit 70 interprets the content of the output data from the data output unit 68 and calculates the indicated position or orientation of the electronic pen 14. For example, if the value of the output data is a relative value indicating the coordinates of the indicated position, the data interpretation unit 70 obtains the absolute value of the indicated position or orientation by adding or multiplying it by an offset value indicating a reference value. As a result, position and orientation information of the electronic pen 14 is obtained.

[0057] <Pen state detection operation> Next, the pen state detection operation by the touch IC 36 in Figure 3 (more specifically, the detection unit 50 in Figures 3 and 4) will be explained with reference to the functional block diagram in Figure 4, the flowchart in Figure 5, and Figures 6 to 11.

[0058] In step SP10 of Figure 5, the detection unit 50 checks whether the detection timing has arrived. If the detection timing has not yet arrived (step SP10: NO), the detection unit 50 remains in step SP10 until that timing arrives. On the other hand, if the detection timing has arrived (step SP10: YES), the detection unit 50 proceeds to the next step SP12.

[0059] In step SP12, the distribution acquisition unit 60 acquires the X signal distribution and the Y signal distribution from the touch sensor 46 through a scanning operation for each sensor electrode 46s. Here, the X signal distribution will be used as an example for explanation.

[0060] Figure 6 is a schematic diagram partially showing the electronic pen 14 shown in Figures 1 and 3. The tip of the electronic pen 14 is coaxially provided with a roughly conical tip electrode 80 and a bottomless frustoconical upper electrode 82. The tip electrode 80 and the upper electrode 82 are pen-side electrodes for outputting signals (so-called downlink signals) generated by the oscillation circuit 84. By changing the oscillation frequency or switching the transmission destination in a time-division manner, the electronic pen 14 can output two types of downlink signals via the tip electrode 80 and the upper electrode 82.

[0061] The touch IC 36 (Figure 3) acquires a signal distribution (hereinafter referred to as the "chip-side signal distribution") from multiple sensor electrodes 46s that shows the change in capacitance as the chip electrode 80 approaches. The chip-side signal distribution typically has a shape with one peak at position Q1. Here, position Q1 corresponds to the position obtained by projecting the top of the chip electrode 80 (position P1) onto the sensor plane.

[0062] Similarly, the touch IC 36 acquires a signal distribution (hereinafter referred to as the "upper side signal distribution") from multiple sensor electrodes 46s that indicates the change in capacitance as the upper electrode 82 approaches. The upper side signal distribution typically has a shape with one or two peaks at position Q2. Here, position Q2 corresponds to the position obtained by projecting the shoulder portion (position P2) of the upper electrode 82 onto the sensor plane. Position Q3, which will be described later, corresponds to the position obtained by projecting the center of the upper bottom surface (position P3) of the upper electrode 82 onto the sensor plane.

[0063] Figure 7 shows an example of a 1D signal distribution acquired from the touch sensor 46. The horizontal axis of the graph represents the line number (i.e., the identification number of the sensor electrode 46s), and the vertical axis represents the signal value. Here, we assume a situation where two electronic pens 14 are detected simultaneously. In this case, the 1D signal distribution shows two narrow peaks centered on the pointing position of the electronic pen 14. On the other hand, at the remaining positions excluding the two peaks, the signal value is 0 or a small value. Hereafter, the entire 1D signal distribution will be referred to as the "overall distribution," and a localized 1D signal distribution where the change in capacitance is relatively large may be referred to as the "local distribution." "Relatively large" may mean that the amount of change is large compared to positions other than the local distribution, or that the amount of change is large compared to a predetermined threshold.

[0064] Figures 8 and 9 show examples of 1D signal distributions detected by the touch sensor 46 when the electronic pen 14 is in contact. More specifically, Figure 8 shows the local distribution on the tip side, and Figure 9 shows the local distribution on the upper side. The horizontal axis of the graph shows the relative position (unit: mm) with respect to the indicated position of the electronic pen 14, and the vertical axis of the graph shows the signal value (unit: none) normalized to [0,1]. The positive and negative signs of these signal values ​​are defined so that they become "positive" when the electronic pen 14 is close. The shape of both signal distributions changes according to the tilt angle of the electronic pen 14. In this figure, three curves obtained by changing the tilt angle are superimposed and shown.

[0065] As shown in Figure 8, the local distribution on the tip side has a generally similar shape regardless of the magnitude of the tilt angle. This is because, while using the electronic pen 14, the top of the tip electrode 80 is usually in the position closest to the sensor plane, and position Q1 roughly coincides with position P1. On the other hand, as shown in Figure 9, the local distribution on the upper side changes significantly in the position or number of peaks depending on the change in tilt angle. This is because, while using the electronic pen 14, one of the shoulders of the upper electrode 82 is usually in the position closest to the sensor plane, and the distance between positions Q1 and Q2 changes according to the tilt angle.

[0066] Using the coordinates of positions Q1 and Q2, the position and orientation (i.e., pen state) of the electronic pen 14 can be estimated. For example, the indicated position corresponds to position Q1 shown in Figure 7. The tilt angle corresponds to the angle θ between the sensor plane and the axis of the electronic pen 14. That is, θ=0° when the pen is horizontal to the sensor plane, and θ=90° when it is perpendicular to the sensor plane.

[0067] In step SP14 of Figure 5, the distribution synthesis unit 62 generates a 2D signal distribution by synthesizing the X signal distribution and Y signal distribution obtained in step SP12. Alternatively, the distribution synthesis unit 62 obtains the X feature distribution related to the X signal distribution and the Y feature distribution related to the Y signal distribution, respectively, and generates a 2D feature distribution by synthesizing the X feature distribution and the Y feature distribution.

[0068] Figure 10 shows an example of a method for combining 1D signal distributions. The first table 90 shows the overall or local distribution in the X direction, and the second table 92 shows the overall or local distribution in the Y direction. The cells in the first table 90 represent positions on the X axis (absolute or relative coordinates), and the cells in the second table 92 represent positions on the Y axis (absolute or relative coordinates).

[0069] In the example of this figure, the distribution synthesis unit 62 is composed of a multiplier that multiplies the "first input value" obtained by referring to the first table 90 and the "second input value" obtained by referring to the first table 90, and outputs these multiplied values. By calculating and arranging the multiplied values for each position on the X-axis and each position on the Y-axis, a map 94 showing a two-dimensional signal distribution is obtained.

[0070] FIG. 11 is a diagram showing an example of a method for calculating a 1D feature amount distribution. As shown in FIG. 11(a), in order from the younger line number, S n-2 = 0.15 / S n-1 = 0.40 / S n = 0.80 / S n+1 = 0.30 / S n+2 Suppose a signal distribution where = 0.10 is obtained. Note that the signal values at other line numbers are either 0 or negligibly small values. For example, according to the following equations (1) and (2), {G i} and {F i} are calculated respectively. G i = (S i - S i-2 ) + (S i-1 - S i-3 ) ···(1) F i = |G i | / max{|G i |} ···(2)

[0071] As a result, the "signed slope" {G i} shown in FIG. 11(b) and the feature amount {F i} shown in FIG. 11(c) are calculated respectively. As understood from equation (2), this feature amount {F i} corresponds to the "unsigned slope" normalized in the range of [0, 1].

[0072] In step SP16 of FIG. 5, the estimator 64 (more specifically, the feature map generation unit 66) generates a feature map for estimating the pen state from the 2D signal distribution synthesized in step SP14.

[0073] In step SP18, the estimator 64 (more specifically, the data output unit 68) generates and outputs data (i.e., output data) that correlates with the instruction and posture of the electronic pen 14 from the feature map generated in step SP16.

[0074] In step SP20, the estimator 64 (more specifically, the data interpretation unit 70) converts the output data output in step SP18 into position and orientation information indicating the position and orientation of the electronic pen 14.

[0075] Subsequently, returning to step SP10, the detection unit 50 sequentially executes steps SP10 to SP20 according to a predetermined order. This allows the indicated position and orientation of the electronic pen 14 to be detected one after another.

[0076] [Specific examples of input / output models 64A to 64D] Next, specific examples of input / output models 64A to 64D for estimating the state of the electronic pen 14 will be explained in detail with reference to Figures 12 to 15. Note that the unfilled circles shown in Figures 12 to 15 correspond to the computing units that make up the neural network.

[0077] <Example 1> Figure 12 shows the network structure that constitutes the first example input / output model 64A. This input / output model 64A is composed of a so-called convolutional neural network (CNN). This input / output model 64A consists of an input layer 100, a group of hidden layers 102, and an output layer 104.

[0078] The input layer 100 consists of Nx·Ny arithmetic units for inputting the 2D signal distribution from the upper side. Nx corresponds to the number of pixels in the X direction, and Ny corresponds to the number of pixels in the Y direction. In the first example, the 2D signal distribution is represented as an image signal with one color channel.

[0079] The intermediate layer group 102 consists of one or more intermediate layers, in the first example, four intermediate layers 102a, 102b, 102c, and 102d connected in series. Each intermediate layer consists of one of the following computational layers: [1] a convolutional layer, [2] a pooling layer, or [3] a normalization layer. The feature map from intermediate layer 104d is flattened (made one-dimensional) prior to being supplied to the output layer 104.

[0080] The output layer 104 consists of two arithmetic units for outputting the position indicated by the electronic pen 14. One arithmetic unit outputs the relative value of the X coordinate, and the other arithmetic unit outputs the relative value of the Y coordinate. The relative value is, for example, a value normalized so that the range from the lower limit to the upper limit is [0,1].

[0081] Thus, the input / output model 64A may be a machine learning model including a convolutional neural network that takes a 2D signal distribution or a 2D feature distribution as input and outputs coordinate values ​​indicating the indicated position of the electronic pen 14. With the configuration of the first example, the indicated position of the electronic pen 14 can be estimated with higher accuracy based on the 2D signal distribution on the upper side.

[0082] <Example 2> Figure 13 shows the network structure that constitutes the second example of input / output model 64B. This input / output model 64B has a different network structure from the first example (input / output model 64A in Figure 12) in that the output side is branched into two.

[0083] Specifically, the input / output model 64B comprises an input layer 110, an intermediate layer group 112, and an output layer group 114. The input layer 110 consists of Nx·Ny arithmetic units for inputting the upper-side 2D signal distribution, similar to the first example (input layer 100 in Figure 12).

[0084] The intermediate layer group 112 consists of multiple intermediate layers; in the second example, it consists of five intermediate layers 112a, 112b, 112c, 112d, and 112e. The first three intermediate layers 112a to 112c are connected in series. On the other hand, the second two intermediate layers 112d and 112e are connected in parallel to the output side of the upstream intermediate layer 112c.

[0085] The output layer group 114 comprises an output layer 114d connected to the intermediate layer 112d and an output layer 114e connected to the intermediate layer 112e. The output layer 114d consists of one arithmetic unit for outputting the "relative value of the X coordinate," which is the indicated position of the electronic pen 14. The output layer 114e consists of one arithmetic unit for outputting the "relative value of the Y coordinate," which is the indicated position of the electronic pen 14.

[0086] Thus, the input / output model 64B may be a machine learning model that includes a convolutional neural network that takes a 2D signal distribution or a 2D feature distribution as input and outputs coordinate values ​​indicating the indicated position of the electronic pen 14. In the second example configuration as well, the indicated position of the electronic pen 14 is estimated with higher accuracy based on the upper 2D signal distribution, similar to the first example.

[0087] <Example 3> Figure 14 shows the network structure that constitutes the third example of input / output model 64C. This input / output model 64C is composed of an input layer 120, an intermediate layer group 122, and an output layer group 124.

[0088] The input layer 120 consists of Nx·Ny arithmetic units for inputting a 2D signal distribution, as in the first and second examples. However, the input signal in the third example is an image signal with two color channels. For example, the first color channel corresponds to the upper-side 2D signal distribution, and the second color channel corresponds to the chip-side 2D signal distribution.

[0089] The intermediate layer group 122 consists of five intermediate layers 122a, 122b, 122c, 122d, and 122e, similar to the second example (intermediate layer group 112 in Figure 13). The two later intermediate layers 122d and 122e are connected in parallel to the output side of the upstream intermediate layer 122c.

[0090] The output layer group 124 comprises an output layer 124d connected to the intermediate layer 122d and an output layer 124e connected to the intermediate layer 122e. The output layer 124d consists of one arithmetic unit for outputting the "relative value of the tilt angle," which is the orientation of the electronic pen 14. The output layer 124e consists of one arithmetic unit for outputting the "relative value of the rotation angle," which is the orientation of the electronic pen 14.

[0091] Thus, the input / output model 64C may be a machine learning model including a convolutional neural network that takes a 2D signal distribution or a 2D feature distribution as input and outputs at least one of the tilt angle and rotation angle that indicate the orientation of the electronic pen 14. With the configuration of the third example, by providing the above-described input / output model 64C, the orientation of the electronic pen 14 can be estimated with high accuracy based on the combination of the 2D signal distributions on the upper side and the tip side.

[0092] <Example 4> Figure 15 shows the network structure that constitutes the input / output model 64D of the fourth example. This input / output model 64D is composed of the input / output model 64C (Figure 14) shown in the third example. Specifically, in addition to the input / output model 64C, the input / output model 64D is composed of two sub-models 130 and 140 that are connected in parallel to the output side of the input / output model 64C.

[0093] Submodel 130 includes a hierarchical neural network comprising an input layer 132, a hidden layer group 134, and an output layer 136. The input layer 132 consists of (Nx+2) computation units for inputting [1] the upper X signal distribution and [2] the angle estimate. The X signal distribution is identical to the distribution that generated the 2D signal distribution input to input / output model 64C. The angle estimate is identical to the tilt angle and rotation angle estimates output from input / output model 64C, respectively.

[0094] The intermediate layer group 134 consists of one or more intermediate layers. The output layer 136 consists of one arithmetic unit for outputting the "relative value of the X coordinate," which is the indicated position of the electronic pen 14.

[0095] Submodel 140, like submodel 130, includes a hierarchical neural network comprising an input layer 142, a hidden layer group 144, and an output layer 146. The input layer 142 consists of (Ny+2) computational units for inputting [1] the upper Y signal distribution and [2] the angle estimate. The Y signal distribution is identical to the distribution that generated the 2D signal distribution input to input / output model 64C. The angle estimate is identical to the tilt angle and rotation angle estimates output from input / output model 64C, respectively.

[0096] The intermediate layer group 144 consists of one or more intermediate layers. The output layer 146 consists of one arithmetic unit for outputting the "relative value of the Y coordinate," which is the indicated position of the electronic pen 14.

[0097] Thus, the input / output model 64D may be a machine learning model that includes the input / output model 64C shown in Figure 14. In other words, a new input / output model 64D may be constructed by connecting submodels 130 and 140 to the input / output model 64C. With the configuration of the fourth example, the orientation and pointing position of the electronic pen 14 are estimated with high accuracy based on a combination of 1D signal distribution and 2D signal distribution, or a combination of 1D feature distribution and 2D feature distribution.

[0098] For example, submodel 130 may be a first-layer neural network that takes at least one of the tilt angle and rotation angle output from input / output model 64C and an X signal distribution or X feature distribution as inputs, and outputs an X coordinate value indicating the indicated position of the electronic pen 14. Alternatively, submodel 140 may be a second-layer neural network that takes at least one of the tilt angle and rotation angle output from input / output model 64C and a Y signal distribution or Y feature distribution as inputs, and outputs a Y coordinate value indicating the indicated position of the electronic pen 14.

[0099] [Summary of Embodiments] As described above, the pen state detection circuit (here, touch IC 36) in this embodiment detects the state of an electronic pen 14 having at least one electrode based on the signal distribution detected by a capacitive touch sensor 46, which has multiple sensor electrodes 46s arranged in a planar manner.

[0100] The touch IC 36 then performs the following steps: an acquisition step (SP12 in Figure 5) to acquire from the touch sensor 46 an X signal distribution along a first direction (X direction) and a Y signal distribution along a second direction (Y direction) intersecting the first direction, which are one-dimensional signal distributions indicating the change in capacitance as the electrodes approach the touch sensor 46; a synthesis step (SP14) to generate a 2D signal distribution by combining the acquired X signal distribution and Y signal distribution, or to generate a 2D feature distribution by combining an X feature distribution relating to the shape of the X signal distribution and a Y feature distribution relating to the shape of the Y signal distribution; and an estimation step (SP16 to SP20) to estimate the instruction position or orientation of the electronic pen 14 from the synthesized 2D signal distribution or 2D feature distribution using predetermined input / output models 64A to 64D.

[0101] With this configuration, the accuracy of estimating the pen state in the electronic pen 14 having at least one electrode can be improved. In particular, since the 2D signal distribution (or 2D feature distribution) is calculated by combining the X signal distribution and the Y signal distribution (or the X feature distribution and the Y feature distribution), the amount of data to be handled can be reduced compared to when acquiring the 2D signal distribution from the touch sensor 46, and the computational processing can be sped up accordingly.

[0102] Alternatively, in the synthesis step, a 2D signal distribution may be generated by multiplying a first corresponding value in the X-signal distribution that corresponds to the position in the X-direction and a second corresponding value in the Y-signal distribution that corresponds to the position in the Y-direction. Or, in the synthesis step, a 2D feature distribution may be generated by multiplying a first corresponding value in the X-signal distribution that corresponds to the position in the X-direction and a second corresponding value in the Y-direction.

[0103] Furthermore, if the electronic pen 14 is provided with multiple electrodes, the 2D signal distribution or 2D feature distribution may consist of an image signal having multiple color channels corresponding to the electrodes. In particular, the electronic pen 14 may be provided with at least a tip electrode 80 having a shape symmetrical to the axis of the electronic pen 14 and provided at the tip of the electronic pen 14, and an upper electrode 82 having a shape symmetrical to the axis of the electronic pen 14 and provided on the proximal end side of the tip electrode 80. This makes it possible to estimate the orientation of the electronic pen 14 (e.g., tilt angle and rotation angle) with high accuracy.

[0104] Furthermore, the input system 10 in this embodiment includes an electronic device 12 configured to include a pen state detection circuit (here, a touch IC 36), an electronic pen 14 used together with the electronic device 12, and a server device 16 that can communicate bidirectionally with the electronic device 12 and is configured to store model parameters 20 that describe input / output models 64A to 64D constructed on the touch IC 36. When the electronic pen 14 is detected, the electronic device 12 may request the server device 16 to transmit the model parameters 20 corresponding to the electronic pen 14. This allows the electronic device 12 to automatically utilize the input / output models 64A to 64D corresponding to the electronic pen 14.

[0105] [Differentiation] It should be noted that the present invention is not limited to the embodiments described above, and can be freely modified without departing from the spirit of the invention. Alternatively, the various components may be combined in any way that does not create a technical inconsistency. Alternatively, the execution order of each step constituting the flowchart may be changed as long as it does not create a technical inconsistency. [Explanation of Symbols]

[0106] 10…Input system, 12…Electronic device (pen state detection device), 14…Electronic pen, 16…Server device, 36…Touch IC (pen state detection circuit), 46…Touch sensor, 50…Detection unit, 64…Estimator, 64A~64D…Input / output model, 80…Chip electrode (electrode), 82…Upper electrode (electrode), 90…First table (first signal distribution), 92…Second table (second signal distribution), 94…2D map (two-dimensional signal distribution)

Claims

1. A pen state detection circuit that detects the state of an electronic pen having at least one electrode based on a signal distribution detected by a capacitive touch sensor having multiple sensor electrodes arranged in a planar manner, A one-dimensional signal distribution showing the change in capacitance associated with the approach of the electrode, comprising an acquisition step of acquiring a first signal distribution along a first direction on the touch sensor and a second signal distribution along a second direction intersecting the first direction from the touch sensor, A synthesis step of generating a two-dimensional signal distribution by combining the acquired first signal distribution and the second signal distribution, or generating a two-dimensional feature distribution by combining a first feature distribution relating to the shape of the first signal distribution and a second feature distribution relating to the shape of the second signal distribution, An estimation step in which the instruction position or orientation of the electronic pen is estimated from the synthesized two-dimensional signal distribution or the two-dimensional feature distribution using a predetermined input / output model, A pen state detection circuit characterized by performing the following actions.

2. In the synthesis step, the two-dimensional signal distribution is generated by multiplying a first corresponding value corresponding to the position in the first direction in the first signal distribution by a second corresponding value corresponding to the position in the second direction in the second signal distribution, or the two-dimensional feature distribution is generated by multiplying a first corresponding value corresponding to the position in the first direction in the first feature distribution by a second corresponding value corresponding to the position in the second direction in the second feature distribution. The pen state detection circuit according to claim 1.

3. The input / output model is a machine learning model that includes a convolutional neural network that takes the two-dimensional signal distribution or the two-dimensional feature distribution as input and outputs coordinate values ​​indicating the position indicated by the electronic pen. The pen state detection circuit according to claim 1.

4. The input / output model is a machine learning model that includes a convolutional neural network that takes the two-dimensional signal distribution or the two-dimensional feature distribution as input and outputs at least one of the tilt angle and rotation angle that indicate the orientation of the electronic pen. The pen state detection circuit according to claim 1.

5. The aforementioned input / output model, A convolutional neural network that takes the aforementioned two-dimensional signal distribution or the aforementioned two-dimensional feature distribution as input and outputs at least one of the tilt angle and rotation angle that indicate the orientation of the electronic pen, A first-layer neural network that takes at least one of the tilt angle and rotation angle output from the convolutional neural network and the first signal distribution or the first feature distribution as inputs and outputs coordinate values ​​in the first direction indicating the position of the electronic pen, A second-layer neural network that takes at least one of the tilt angle and rotation angle output from the convolutional neural network and the second signal distribution or the second feature distribution as inputs and outputs coordinate values ​​in the second direction indicating the position of the electronic pen, The pen state detection circuit according to claim 1, which includes a machine learning model.

6. The electronic pen is provided with a plurality of electrodes, The two-dimensional signal distribution or the two-dimensional feature distribution consists of an image signal having multiple color channels corresponding to the electrodes. The pen state detection circuit according to claim 1.

7. The electronic pen is provided with at least a tip electrode having a shape symmetrical to the shaft of the electronic pen and provided at the tip of the electronic pen, and an upper electrode having a shape symmetrical to the shaft of the electronic pen and provided on the proximal end side of the tip electrode. The pen state detection circuit according to feature 6.

8. A pen state detection method for detecting the state of an electronic pen having at least one electrode, based on a signal distribution detected by a capacitive touch sensor having multiple sensor electrodes arranged in a planar manner, A one-dimensional signal distribution showing the change in capacitance associated with the approach of the electrode, comprising an acquisition step of acquiring a first signal distribution along a first direction on the touch sensor and a second signal distribution along a second direction intersecting the first direction from the touch sensor, A synthesis step of generating a two-dimensional signal distribution by combining the acquired first signal distribution and the second signal distribution, or generating a two-dimensional feature distribution by combining a first feature distribution relating to the shape of the first signal distribution and a second feature distribution relating to the shape of the second signal distribution, An estimation step in which the instruction position or orientation of the electronic pen is estimated from the synthesized two-dimensional signal distribution or the two-dimensional feature distribution using a predetermined input / output model, A pen state detection method characterized by one or more processors executing the following.

9. An electronic device comprising the pen state detection circuit described in claim 1, An electronic pen used together with the aforementioned electronic device, A server device that is capable of bidirectional communication with the aforementioned electronic device and is configured to store model parameters that describe the input / output model constructed in the pen state detection circuit. Equipped with, The input system is characterized in that, when the electronic device detects the electronic pen, it requests the server device to transmit model parameters corresponding to the electronic pen.

Images