Digital stylus prediction by gesture

Abstract

The methods disclosed herein include determining a user interaction pattern with respect to a digital stylus, the user interaction pattern based at least in part on user interaction with a capacitive sensor of a computing device; determining a geometric characteristic of the digital stylus; and generating a predicted digital mark location of the digital stylus on a digital medium of the computing device based at least in part on the determined geometric characteristic of the digital stylus and the determined user interaction pattern.

Description

Background Technology

[0001] Writing with digital markers such as styluses can encounter a delay between the marker's touch sensing and the actual marking. This can be frustrating for users, who may rewrite elements based on the assumption that the delayed marking indicates a missing mark. Summary of the Invention

[0002] The method disclosed herein includes: determining a user interaction pattern for a digital marker, the user interaction pattern being at least partially based on user interaction with a capacitive sensor of a computing device; determining geometric features of the digital marker; and generating a predicted digital mark position of the digital marker on a digital medium of the computing device based at least partially on the determined geometric features of the digital marker and the determined user interaction pattern.

[0003] The described techniques provide implementations of systems and methods for digital tagging. More specifically, the described techniques provide implementations of systems and methods for predictive digital tagging.

[0004] This summary is provided to introduce, in a simplified form, some concepts that will be further described in the following detailed embodiments. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0005] This article also describes and lists other implementations. Attached Figure Description

[0006] Figure 1 An example system for predicting the location of digital markers is shown.

[0007] Figure 2 Another example system for predicting the location of digital markers is shown.

[0008] Figure 3 An example operation for generating the predicted tags is shown.

[0009] Figure 4 An example operation for generating the predicted marker positions is shown.

[0010] Figure 5 An example implementation of memory 500 is shown.

[0011] Figure 6 An example computing device is shown that is used to implement the features and operations of the described technology. Detailed Implementation

[0012] Writing with a digital marker offers significant advantages over relying on typical mouse or touchpad systems, but latency can be a concern. One way to improve the user experience is to generate predictive marks before the digital marker touches the digital marking medium. Considering the user's interaction with the device's sensors, in addition to the sensors within the digital marker itself, can provide a predictive advantage. Predicting the potential location of the mark on the device's digital medium can be difficult and can complicate device design. For example, each additional sensor is an expensive component, and relying on digital marker sensors, which have limited applications, is problematic. A digital marker may be able to track its position, orientation, and speed, but it doesn't necessarily indicate any information about the user's characteristics. Furthermore, expecting users to use equipment that is not traditionally deployed may be unrealistic. Utilizing different aspects of devices that are necessary for or typically already provide digital marking media to improve mark prediction can be advantageous.

[0013] Devices with digital tagging media can interact with digital markers to receive digital tags. The digital tagging media can also be a user interaction sensor, such as a capacitive sensor or a touch sensor. The user interaction sensor can be used to receive active or passive signals from the digital marker.

[0014] In addition to interactions generated by the digital marker, user interaction sensors can also interact with the user. When writing with a digital marker on a digital medium (such as a touch-sensitive screen of a computing device), the user interaction sensor can sense interactions with the user's anatomical features as the user holds the digital marker. For example, the user interaction sensor can sense one or more of the following: direct touch patterns of anatomical features, proximity patterns of anatomical features, and shadow patterns projected by anatomical features.

[0015] Using user-interactive sensors and digital marker sensors to predict potential marker locations offers several advantages. These sensors can transmit data at a lower level of hardware, closer to the processor, allowing for faster processing and prediction. This synergy reduces latency in the predictive model compared to systems requiring further hardware or higher-level processing.

[0016] Based on the sensor data, the device system can infer user interaction patterns, which may include characteristics of the user's interaction with the marker device and / or the user's marking-related characteristics. Sensor data may be used to determine user interaction patterns, such as one or more of the user's dominant hand, hand orientation, and hand size.

[0017] These user interaction patterns can be further used to identify predictable user characteristics that can inform tagging predictions, such as age, handwriting style, medical issues affecting handwriting, gender, athletic ability, or handwriting. While the default algorithm can initially be used for predictive tagging, user characteristics can be used to further refine the predictive model and even customize the predictive model for specific users over a period of time.

[0018] Predictive models can generate a probability distribution of a reliable digital marker state to predict the most likely location of the marker. The probability distribution can include the uncertainty of the digital marker state or be derived from that uncertainty. The digital marker state can include the geometric features of the digital marker, such as its orientation or position relative to the digital marking medium. The probability of a particular marker state can be indicated by various data. Data from the digital marker itself can include, for example, its geometric features and its velocity.

[0019] The predictive model can rely on hysteresis to determine possible marker locations. The probability distribution of digit marker states from previous predictive iterations can be used to inform the predictive model. In one implementation, the current user interaction pattern can be determined based on sensed user interaction with the digit marker medium. Based on these patterns, the inference engine can generate a first probability distribution of digit marker states. This first probability distribution can be combined with previous probability distributions of digit marker states to generate a second probability distribution, thereby reducing the number of digit marker states used to predict the markers. This can be done, for example, by using convolutions of the distributions. The second probability distribution can be further transformed and combined with current data, such as current data related to sensed user interactions and / or the geometric features of the digit markers, to further reduce the number of potential digit marker states in a third probability distribution. This can be done using an inference engine and can utilize filters such as Kalman filters or other state machines. Based on this third probability distribution of marker states, the inference engine can predict the predicted marker location. This method can provide a significantly reduced set of possible digital marker states compared to systems that do not consider user-to-user interaction sensors, thereby reducing the range of potential marker locations.

[0020] The predicted marker position generated based on the probability distribution of the marker's state can also be further reconciled with predictions generated by other models. For example, the operating system's inference engine can use predictive modeling that does not rely on digital marker sensor data or only on existing sensor data to predict marker positions. These processes may be faster because they depend on the operating system to the software stack, but they can be affected by many assumptions and can be better developed by taking into account currently sensed data from the digital marker and user interaction sensors. The inference engine can combine this data to generate a more robust predicted marker position.

[0021] Figure 1 An example system 100 for predicting the location of digital markers is shown. System 100 has a device 102 with a user interaction sensor 104. This device may be an implementation of a computing device 500. In the illustrated implementation, the user interaction sensor 104 is a capacitive sensor. A user 108 interacts with a digital marker 106 to make a mark 112. When making the mark 112, the user 108 also interacts with the user interaction sensor 104. The interaction with the user interaction sensor 104 may be characterized, for example, by touch patterns 130 and / or shadows 132 of the anatomical features of the user 108. Anatomical features may be appendages, such as the user 108's arm and / or hand.

[0022] User interactions can indicate user interaction patterns, such as the user's dominant hand, user interaction orientation, or the size of the user's hand; the inference engine can determine the user interaction pattern based at least in part on the interaction with the user interaction sensor 104. The user's dominant hand can be an indication of whether the user 108 is right-handed, left-handed, or ambidextrous. The user interaction orientation can originate from a viewpoint, such as from the position on the user interaction sensor 104 and / or the axis of the user interaction sensor 104, from the position on the digital marker and / or orientation, or from the position established relative to the user interaction dynamics (e.g., from the angle of an element of touch pattern 130 or shadow 132). The user's dominant hand may affect hand posture and may limit the potential area that can be marked relative to the state of the digital marker.

[0023] Determining the user interaction pattern can help reduce the number of plausible locations for the predicted marker. As shown, user 108 is right-handed and has a bent hand posture, with part of the hand resting on user interaction sensor 104. Based on this posture, there is a finite reach area 110 for the digital marker 106. By considering the finite reach area 110 using the user interaction pattern, the predicted marker location can be restricted to a position within the finite reach area 110.

[0024] As shown in the figure, user 108 is attempting to draw a "3" using a digital marker 106. The fast prediction model in the operating system of device 102 faces greater difficulty in predicting curved markings, such as the curved marks required to write a "3". Predictive models can benefit from information about user interaction patterns.

[0025] In one implementation, data generated from the digital marker 106 can be used to determine the geometric features of the digital marker 106. The digital marker 106 may have one or more sensors, such as an accelerometer, gyroscope, capacitive sensor, motion sensor, tilt sensor, device-specific digitizer transceiver, and pressure sensor. The digital marker 106 can use the sensors to determine or transmit data reflecting two-dimensional and / or three-dimensional geometric features, such as one or more of the orientation, position, speed of movement, tilt, orientation, area, or element of the digital marker 106 interacting with the user 108, and possible changes in these geometric features relative to the user interaction sensor 104. The digital marker state may include data about one or more geometric features of the digital marker 106, which may be related to the predicted marker position.

[0026] In one implementation, data generated from the user interaction sensor 104 can be used to determine user interaction patterns with respect to the digital marker 106. The user interaction sensor 104 may include one or more interaction sensors, such as capacitive sensors, motion sensors, accelerometers, click sensors, light sensors, and temperature sensors. The interaction sensors can sense user interaction with the user interaction sensor 104, for example, by sensing one or more touch patterns 130 and / or shadows 132 projected from the anatomical features of the user 108. The nature of the physical user interaction and shadow projection can help indicate the user interaction pattern, which can improve the model used to determine the predicted marker location. The data generated by the interaction sensor can provide information about the interaction pattern, such as one or more of the user's dominant hand, user hand size, and user hand orientation. The interaction pattern can be determined relative to the user interaction sensor 104, device 102, and / or digital marker 106. Based on the user interaction pattern, an inference engine can determine possible marker locations and / or possible hand postures. Figure 1As shown, the sensed hand pose can be used to limit the area of ​​potential markers to a limited reachable region 110. The inference engine can also determine the characteristics and / or classification of user 108 based on user interaction patterns, such as age, handwriting style, medical problems affecting handwriting, gender, motor ability, or one or more of handwriting characteristics. While the default algorithm can initially be used for predictive markers, the characteristics of user 108 can be used to further refine the predictive model and even customize the model for user 108 over a period of time. Considering user characteristics allows the predictive model to better predict the next marker, at least in part, based on one or more of past user behavior and behaviors indicating user type.

[0027] The inference engine can use derived user interaction patterns and other sensed user interaction data to generate potential digit marker state data 106, along with associated probabilities and / or uncertainties for these states. The inference engine can generate a first digit marker state probability distribution based on user interaction data, user interaction patterns, and / or user pose. Because user interaction data can indicate an individual's pose and / or characteristics, it can constrain the expected potential digit marker states, thereby constraining the predicted marker location. Using a digit marker state probability distribution better accounts for uncertainty in the data compared to using a standard probability map without accompanying probabilities or uncertainties.

[0028] The inference engine can combine or otherwise merge a first digital marker state probability distribution with a previously determined previous digital marker state probability distribution to further limit the potential digital marker states used for predicting markers. For example, the inference engine can perform a convolution of the digital marker state probability distributions. This operation produces a second digital marker state probability distribution, which represents a further constraint on the potential digital marker states, and thus a further constraint on the potential locations of the predicted markers. Considering the previous digital marker state probability distributions can narrow the range of possible predicted marker locations because, in the time increments between markers, the previous marker data will indicate the next marker. For the purposes of this specification, when data is combined or merged, it should be understood that this can involve one or more operations such as convolution, linear transformation, machine learning inference, state machine processing, Kalman filter processing, and processing through other inference methods. Combining or merging can cause an overall change in the number of digital marker states or marking positions, can cause a weighted probability of some of the overlapping or intersecting digital marker states or marking positions, or can cause a weighted probability of marker states in which no marker state is eliminated as a probability from the probability distribution of the first and previous marker states.

[0029] The inference engine can further transform the probability distribution of the second digital marker state. This transformation allows for better use of data in the state model used to predict marker positions. In this transformation, the first state can be used to predict the current state from previous data using Kalman state model parameters (e.g., velocity, orientation, and consistency in ink prediction). This can be done using, for example, matrix multiplication. An example matrix multiplication is between a two-dimensional or three-dimensional position matrix (potentially accompanied by uncertainty in each dimension) and a two-dimensional or three-dimensional velocity matrix (potentially accompanied by uncertainty and potentially further considering directional scattering data). The next step in this transformation could be combining the predicted Gaussian probability distribution with sensor data (e.g., position and signal-to-noise ratio sensors). The second step could include matrix multiplication between the matrix obtained from the previous step and a second matrix representing the two-dimensional or three-dimensional position sensor and signal-to-noise ratio sensor data. The result of this transformation could be a Gaussian probability distribution, where its center is the most likely predicted digital marker position.

[0030] The inference engine can use the transformed second digital marker state probability distribution with the current digital marker 106 sensor data and user interaction sensor 104 data to generate a third digital marker state probability distribution. Considering the current sensor data, the sensor data from the digital markers and / or user interaction sensors may provide a more comprehensive picture of the possible marker states, and therefore a more comprehensive picture of the potential marker locations. The inference engine can use a state machine or a filter, such as a Kalman filter, to combine this data. The current digital marker 106 sensor data and user interaction sensor 104 data can be directly input, or they can be processed by the inference engine to generate potential digital marker states, the potential probabilities associated with these digital marker states, and / or the current digital marker state probability distribution.

[0031] The third digit marker state probability distribution can represent the probability of any particular marker state that is at least partially reliable based on data ingested by the inference engine. The inference engine can predict the digit marker position based at least partially on the third digit marker state probability distribution. In one implementation, the inference engine predicts the marker position based on the most probable digit marker state. In another implementation, the inference engine can predict the marker position by determining the marker position probability distribution based on the third digit marker state probability distribution and predicting the digit marker position based on the marker position probability distribution.

[0032] The relationship between the predicted marker position and the digit marker state can be based at least in part on the geometric features of the digit marker 106. For example, the orientation and position of the digit marker 106 can only represent a specific region on which the digit marker 106 can make digit marks. An inference engine that restricts these potential digit marker states and evaluates the probability of each of these states can significantly restrict this region and generate highly probable positions for the markers. In one implementation, the digit marker position can be derived from the marker state by determining the centroid on each axis of consideration and calculating the signal-to-noise ratio (possibly using the phase difference on the marker position electrodes on each axis).

[0033] The inference engine can preemptively mark at the predicted digital marker position before considering other data. Alternatively or supplementarily, the inference engine can combine data generated by the operating system or by the device 102 using a software-based predictive model, or by other components of the inference engine, with data generated from one or more of the third digital marker state probability distribution, the marker position probability distribution, and the predicted marker position, to produce a valid predicted digital marker position. These software-based predictive models may be independent of sensor data or may rely on older sensor data to determine reasonable predictions. These software-based predictive models can operate faster and / or more frequently and can provide predicted digital marker positions or data to be combined with sensor-based data. The software-based predicted position can be considered a “wet ink” mark, which will not ultimately be determined as a “dry ink” mark until the sensor-based data is combined or otherwise considered. Combining the sensor-based predicted position data with the software-based predicted position data produces a validated predicted digital marker position. The probability distribution of the third digital marker state generated at the nearest point can be used as the probability distribution of the previous digital marker state in the next or a later iteration.

[0034] An alternative implementation is envisioned, in which the digital marker state probability distribution is used to generate the marker position probability distribution before being merged or combined, allowing the inference engine to produce the marker position probability distribution at each step. In this implementation, the inference engine can use the first digital marker state probability distribution to determine the first marker position probability distribution. The inference engine can combine the first marker position probability distribution with the previous marker position probability distribution to produce a second marker position probability distribution. The inference engine can acquire current digital marker 106 sensor data and user interaction sensor 104 data, and combine them to determine one or more of the sensor digital marker state probability distribution and the sensor marker position probability distribution. The inference engine can combine the second marker position probability distribution with one or more of the current sensor digital marker state probability distribution and the current sensor marker position probability distribution. These distributions can be combined using one or more of convolution, linear transformation, state machine, machine learning implementation, Kalman filter, and other inference methods. Based on this combination, the inference engine can produce a third marker position probability distribution. Based on the probability distribution of the third marker location, the inference engine can directly predict the marker location, or it can combine the predicted location or the probability distribution of the third marker location with other publicly disclosed software-based predictions to predict the marker location.

[0035] Figure 2 Another example system 200 for predicting the location of a digital marker is shown. User interaction sensor data can be provided by a user interaction sensor. The user interaction sensor data can be used to determine a first digital marker state probability distribution 220. As shown, the first marker state probability distribution 220 represents the distribution of the digital marker state probabilities.

[0036] The previous digit marker state probability distribution 222 can be incorporated into system 200 to account for the possibility that the previous digit marker state probability distribution 222 will affect the position of the next digit marker. A distribution combiner 260 can be used to combine or merge the first marker state probability distribution 220 with the previous digit marker state probability distribution 222. The distribution combiner 260 can be a component of the inference engine of a computing device. The distribution combiner 260 can combine the data by, for example, linear transformation, convolution of the digit marker state probability distributions 220 and 222, combination and weighting of marker state probabilities, and / or applying relative weighting only to marker states present in both the digit marker state probability effect distributions 220 and 222. The distribution combiner 260 generates a second digit marker state probability distribution 224 based on the first digit marker state probability distribution 220 and the previous digit marker state probability distribution 222. In each implementation, the second digital marker state probability distribution represents fewer marker states than the previous digital marker state probability distribution or the first digital marker state probability distribution.

[0037] The marker position predictor 262 can then combine or merge the second digital marker state probability distribution 224 with the current sensor data 228 and predict the marker position based on this combination. The marker position predictor 262 can be an element of an inference engine. In the illustrated implementation, the marker position predictor 262 can initially transform the second digital marker state probability distribution 224 before combining or merging to generate a predicted or transformed second digital marker state probability distribution 226. For example, in this transformation, a first state can be used to predict the current state from previous data using Kalman state model parameters (e.g., velocity, orientation, and consistency in ink prediction). This can be done using, for example, matrix multiplication. An example matrix multiplication is between a two-dimensional or three-dimensional position matrix (possibly accompanied by uncertainty in each dimension) and a two-dimensional or three-dimensional velocity matrix (possibly accompanied by uncertainty and possibly further considering directional scattering data). The next step in this transformation can be to combine the predicted Gaussian probability distribution with sensor data (e.g., position and signal-to-noise ratio sensors). The second step may include matrix multiplication between the matrix obtained from the previous step and a second matrix representing the data from the two-dimensional or three-dimensional position sensor and signal-to-noise ratio sensor. The result of this transformation may be a Gaussian probability distribution, where the center is the most likely predicted digital marker position. In this implementation, the predicted or transformed second digital marker state probability distribution 226 can be combined with the current sensor data 228. An implementation is also envisioned where the marker position predictor 262 does not transform the second digital marker state probability distribution 224, and the second digital marker state probability distribution 224 is combined with the current sensor data. Combination 230 can be processed by the marker position predictor 262 to produce a third digital marker state probability distribution 232.

[0038] In various implementations, the current sensor data 228 may include both digital marker data and user interaction sensor data. As previously stated, the first digital marker state probability distribution is based at least in part on the current user interaction data. For the purposes of this specification, when the term "current" is used, it may mean that the data comes from the same time or sample, or that the various times or samples are close in time. Furthermore, if a previous operation used the current sensor data, and a later operation uses the current sensor data, then the later current sensor data may be the same data, or it may be data currently generated at the time of each operation.

[0039] The marker location predictor 262 may include one or more of the following: Kalman filter, Kalman smoother, α-β filter, inverse variance weighting, covariance intersection, stochastic relation, state transition matrix, kernel adaptive filter, recursive least squares filter, particle filter estimator, moving time-domain estimator, recursive Bayesian estimator, and predictor-corrector method. The Kalman filter may include one or more of the following: frequency-weighted Kalman filter, extended Kalman filter, unscented Kalman filter, discriminative Kalman filter, adaptive Kalman filter, Kalman-Busch filter, and hybrid Kalman filter.

[0040] In the implementation of the marker position predictor 262, which includes a Kalman filter, this combination can use a state model in the prediction step to transform the second digital marker state probability distribution 224 to produce a predicted or transformed second digital marker state probability distribution 226. The relationship between uncertainty and probability can be that the greater the uncertainty, the wider the range of plausible results (e.g., a wider Gaussian probability distribution). This transformation can be generated using a state model. Measurements can be taken using sensors (e.g., digital marker sensors and / or user interaction sensors) to generate current sensor data 228. The current sensor data 228 can be formatted to match the format of the state data in the predicted or transformed second digital marker state probability distribution 226.

[0041] The current sensor data 228 can be formatted to include noise from the measurement. In one implementation, the noise from the measurement can be represented as a probability distribution, possibly a Gaussian and / or independent probability distribution, and / or can be modeled as a measurement with additional, definite uncertainties. This data can be derived from statistical methods utilizing historical measurement uncertainties or noise data. Depending on the implementation, the predicted or transformed second digital marker state probability distribution 226 can be combined with or compared to the current sensor data 228 to update the model. The updated model can produce a prediction of the state variable, which is a third digital marker state probability distribution 232. Based on the third digital marker state probability distribution 232, the predicted marker position based on the sensor can be generated.

[0042] In an alternative embodiment, instead of using and comparing the probability distributions of individual digital marker states in this process, states can be used to represent a reasonable probability distribution of marker positions, where each marker position is a state variable that produces the predicted marker position. In one implementation, the digital marker position can be derived from the pen state by determining the centroid on each axis of consideration and calculating the signal-to-noise ratio (possibly using the phase difference on the pen position electrodes on each axis).

[0043] After generating the predicted marker location based on sensors, a marker can be made on the display at the predicted location where a digital marker is expected to touch the display to generate the marker. Alternatively or supplementarily, the inference engine can further compare the predicted marker location with a software-based marker location prediction generated by a computer system, which does not rely on the most recent sensor readings or any possible sensor readings. For example, another software-based predicted marker location can be generated based at least in part on knowledge of previous marking and the user's behavior in operating the digital marker. In this implementation, potential marker locations can be reconciled from the marker locations predicted by sensor data and other marker locations by, for example, taking the average of the location coordinates and generating the resulting average coordinates. While both sensor-based and software-based predicted marker locations can be based at least in part on one or more of sensor data and software processing, this specification distinguishes between sensor-based predicted marker locations and locations predicted using current and previous sensor data to determine a probabilistic model and generate sensor-based predictions. Software-based predicted marker locations are locations generated without immediate access to sensor data. Combining software-based digital marker position prediction with sensor-based digital marker position prediction can produce validated digital marker position predictions.

[0044] Figure 3 Example operation 300 for generating the predicted mark is shown. Operation 302 determines the user interaction pattern with respect to the digital marker. The user interaction pattern may be based at least in part on user interaction with the device's capacitive sensors. For example, a user may interact with the digital marker to make a mark. The user may interact with the user interaction sensor while making the mark. Interaction with the user interaction sensor may be characterized, for example, by touch patterns and / or shadow projections of the user's anatomical features. Anatomical features may be appendages, such as the user's arm and / or hand.

[0045] User interactions can indicate user interaction patterns, such as the user's dominant hand, user interaction orientation, or user hand size. The inference engine can determine user interaction patterns, at least in part, based on interactions with user interaction sensors. The user's dominant hand can indicate whether the user is right-handed, left-handed, or ambidextrous. The user interaction orientation can originate from a reference or viewpoint, such as from the position or axis on the user interaction sensor, from the position or orientation on the digital marker, or from a position dynamically established relative to the user interaction. The user's dominant hand may influence hand posture and limit the potential area that can be marked relative to the state of the digital marker.

[0046] Determining the user interaction pattern in operation 302 can help reduce the number of plausible locations for the predicted marker. A user may have a hand posture with part of their hand resting on the user interaction sensor. Based on this posture, there exists a finite reach area for the digital marker. By considering this finite reach area using the user interaction pattern, the predicted marker location can be limited to a position within that finite reach area. Fast prediction models in the device's operating system have greater difficulty predicting certain patterns (e.g., curved markers). Predictive models can benefit from information about the user interaction pattern.

[0047] In one implementation, data generated from a user interaction sensor can be used to determine user interaction patterns with respect to a digital marker. The user interaction sensor may include one or more of the following interaction sensors: a capacitive sensor, a motion sensor, an accelerometer, a click sensor, a light sensor, and a temperature sensor. The interaction sensor can sense user interaction with the sensor, for example, by sensing one or more touch patterns and / or shadows projected by the user's anatomical features. The data generated by the interaction sensor can provide information about the interaction pattern, such as one or more of the user's dominant hand, hand size, and hand orientation. The interaction pattern can be determined relative to the user interaction sensor, the device, and / or the digital marker. Based on the user interaction pattern, an inference engine can determine possible marking locations and / or possible hand postures. The sensed hand posture can be used to limit the area of ​​potential marking to a limited reachable area. The inference engine can also determine user characteristics and / or classifications based on the user interaction pattern, such as one or more of age, handwriting style, medical problems affecting handwriting, gender, motor ability, or handwriting style. While the default algorithm can initially be used for predictive labeling, user characteristics can be used to further refine the predictive model, and even to customize the model for the user over time.

[0048] Operation 304 determines the geometric features of the digital marker. In one implementation, data generated from the digital marker can be used to determine the geometric features of the digital marker. The digital marker may have one or more sensors, such as an accelerometer, gyroscope, capacitive sensor, motion sensor, tilt sensor, device-specific digitizer transceiver, and pressure sensor. The digital marker can use the sensors to determine or transmit data reflecting two-dimensional and / or three-dimensional geometric features, such as one or more of the orientation, position, speed of movement, tilt, orientation, area, or element of the digital marker interacting with the user, and possible changes in these geometric features relative to the user-interacting sensors. The state of the digital marker may include data about one or more geometric features of the digital marker, which may be related to the predicted marker position.

[0049] Generation operation 306 generates the predicted digit marker position of the digit marker based at least in part on the determined geometric features of the digit marker and the determined user interaction pattern. Generation operation 306 may be an implementation of operation 400. Generation operation 306 may use an inference engine to derive the user interaction pattern and other sensed user interaction data to generate potential digit marker state data and associated probabilities and / or uncertainties for these states. The inference engine may generate a first digit marker state probability distribution based on the user interaction data, user interaction pattern, and / or user pose. Because the user interaction data can indicate the pose and / or features of an individual, it can constrain the expected potential digit marker states, thereby constraining the predicted marker position. User interaction sensor data may be provided by a user interaction sensor. The user interaction sensor data may be used to determine the first digit marker state probability distribution. The first marker state probability distribution represents the distribution of digit marker state probabilities based at least in part on the user interaction pattern.

[0050] Generation operation 306 can use an inference engine to combine or otherwise merge the first digit marker state probability distribution with a previously determined previous digit marker state probability distribution to further limit the potential digit marker states used for predicting markers. The previous digit marker state probability distribution can be incorporated into the system to account for the possibility that it will influence the next marker position. A distribution combiner can be used to combine or merge the first marker state probability distribution with the previous digit marker state probability distribution. The distribution combiner generates a second digit marker state probability distribution based on the first and previous digit marker state probability distributions.

[0051] Generation operation 306 can also use an inference engine to further transform the second digital marker state probability distribution. In this transformation, the first state can be used to predict the current state from previous data using Kalman state model parameters (e.g., velocity, orientation, and consistency in ink prediction). This can be done using, for example, matrix multiplication. An example matrix multiplication is between a two-dimensional or three-dimensional position matrix (possibly accompanied by uncertainty in each dimension) and a two-dimensional or three-dimensional velocity matrix (possibly accompanied by uncertainty and possibly further considering directional scattering data). The next step in this transformation can be to combine the predicted Gaussian probability distribution with sensor data (e.g., position and signal-to-noise ratio sensors). The second step can include matrix multiplication between the matrix obtained from the previous step and a second matrix representing the two-dimensional or three-dimensional position sensor and signal-to-noise ratio sensor data. The result of this transformation can be a Gaussian probability distribution, where the center is the most likely predicted digital marker position. The inference engine can use the transformed second digital marker state probability distribution with the current digital marker sensor data and user interaction sensor data to generate a third predicted digital marker state probability distribution, at least in part, based on existing and potentially trained state models. The inference engine can use state machines or filters, such as Kalman filters, to combine and / or transform the data.

[0052] In an alternative embodiment, instead of using and comparing the probability distributions of each digital marker state in the process, the state can be used to represent a reasonable probability distribution of marker positions, where each marker position is a state variable that produces the predicted marker position.

[0053] The third digit marker state probability distribution can represent the probability of any particular marker state that is at least partially reliable based on data ingested by the inference engine. The inference engine can predict the digit marker position at least partially based on the third digit marker state probability distribution. In various implementations, the inference engine can compute the current state probability and extract the most probable position from it. In one implementation, the inference engine predicts the marker position based on the most probable digit marker state. In another implementation, the inference engine can predict the marker position by determining the marker position probability distribution based on the third digit marker state probability distribution and predicting the digit marker position based on the marker position probability distribution. In implementations using a Kalman filter, the Kalman filter can produce a Gaussian probability distribution of the digit marker position, where the center of the Gaussian probability distribution (e.g., the position with the highest probability and / or the lowest uncertainty) is the predicted marker position. The relationship between the predicted marker position and the digit marker state can be at least partially based on the geometric features of the digit marker. For example, the orientation and position of the digit marker can only represent specific regions on which the digit marker can make digit markings. An inference engine that limits these potential digital marker states and evaluates the probability of each of these states can significantly limit the area and generate highly probable locations for the markers.

[0054] In one implementation, generation operation 306 can use an inference engine to combine data generated from one or more of the third digital marker state probability distribution, marker position probability distribution, and / or predicted marker position with data generated by an operating system, a device using a software-based prediction model to predict the position, or other components of the inference engine. Combining sensor-based predicted position data with software-based predicted position data can produce more robust predicted digital marker positions.

[0055] The presentation operation 308 presents the predicted mark on the display. The predicted mark can be presented or displayed before the user touches the user interaction sensor with a digital marker. The presentation operation 308 can use an inference engine to preemptively mark at the predicted digital mark location before considering other data or receiving the digital marker touch on the sensor.

[0056] As an alternative or supplement, presentation operation 308 can combine predicted positions generated by different methods by averaging the coordinates of predicted markers generated by different models, taking the median of the coordinates, sorting and selecting the best predicted marker positions, or using one or more consensus mechanisms to determine the predicted markers. The probability distribution of the third digital marker state generated most recently can be used as the probability of the previous digital marker state in the next or later iteration.

[0057] Figure 4An example operation 400 for generating the predicted marker location is shown. Operation 400 may be an example implementation of generation operation 306. Determination operation 402 determines a first marker probability state based at least in part on sensed user interaction patterns. Determination operation 402 may use an inference engine to derive user interaction patterns and other sensed user interaction data to generate potential digital marker state data and associated probabilities and / or uncertainties for these states. Determination operation 402 may use an inference engine to generate a first digital marker state probability distribution based on user interaction data, user interaction patterns, and / or user pose. Because user interaction data can indicate the pose and / or characteristics of an individual, it can constrain the expected potential digital marker states, thereby constraining the predicted marker location. User interaction sensor data may be used to determine the first digital marker state probability distribution. The first marker probability state distribution represents a distribution of digital marker state probabilities based at least in part on user interaction patterns.

[0058] Generation operation 404 generates a second digit marker state probability distribution based at least in part on a first digit marker state probability distribution and a previous digit marker state probability distribution. Generation operation 306 may use an inference engine to combine or otherwise merge the first digit marker state probability distribution with the previously determined previous digit marker state probability distribution to further define the potential digit marker states for predicting markers. The previous digit marker state probability distribution may be incorporated into the system to account for the possibility that the previous digit marker state probability distribution will influence the next marker position. A distribution combiner may be used to combine or merge the first marker state probability distribution with the previous digit marker state probability distribution. The distribution combiner may be a component of the inference engine of a computing device. The distribution combiner may combine the data by, for example, linear transformations, convolutions of the sum of the digit marker state probability distributions, combinations and weights of the marker state probabilities, and / or applying relative weights only to the marker states present in the respective digit marker state probability effect distributions. The distribution combiner generates a second digital marker state probability distribution based on the first digital marker state probability distribution and the previous digital marker state probability distribution.

[0059] For the purposes of this specification, when operating on 400 combined or merged data, this may include one or more operations such as convolution, linear transformation, machine learning inference, state machine processing, Kalman filter processing, and processing by other inference methods. Combining or merging may, for example, cause an overall change in the number of digital marker states or marker positions, cause a weighted average of the probabilities of some overlapping or intersecting digital marker states or marker positions, or cause a weighted probability of marker states where no marker state is eliminated as a probability from the probability distribution of the first and previous marker states.

[0060] Generation operation 406 generates a third digital marker state probability distribution based at least in part on one or more of the second digital marker state probability distribution, sensor data, and marker geometric features. Generation operation 406 may also use an inference engine to further transform the second digital marker state probability distribution. The inference engine can use the transformed second digital marker state probability distribution with the current digital marker sensor data and user interaction sensor data to generate a third predicted digital marker state probability distribution based at least in part on existing and potentially trained state models. The inference engine can use state machines or filters, such as Kalman filters, to combine and / or transform the data. The current digital marker sensor data and user interaction sensor data can be directly input or processed by the inference engine to generate potential digital marker states, potential probabilities and / or uncertainties associated with these digital marker states, and one or more of the third digital marker state probability distribution.

[0061] The marker location predictor may include one or more of the following: Kalman filter, Kalman smoother, α-β filter, inverse variance weighting, covariance intersection, stochastic relation, state transition matrix, kernel adaptive filter, recursive least squares filter, particle filter estimator, moving-time domain estimator, recursive Bayesian estimator, and predictor-corrector method. The Kalman filter may include one or more of the following: frequency-weighted Kalman filter, extended Kalman filter, unscented Kalman filter, discriminative Kalman filter, adaptive Kalman filter, Kalman-Busch filter, and hybrid Kalman filter.

[0062] In an implementation of the marker position predictor that includes a Kalman filter, this combination can use a state model in the prediction step to transform the second digital marker state probability distribution to produce a predicted or transformed second digital marker state probability distribution. Measurements can be taken using sensors (e.g., digital marker sensors and / or user interaction sensors) to generate current sensor data. The current sensor data can be formatted to match the format of the state data in the predicted or transformed second digital marker state probability distribution. The current sensor data can be formatted to include noise from the measurements. In one implementation, the noise from the measurements can be represented as a probability distribution, possibly a Gaussian or independent probability distribution, and / or can be modeled as measurements with accompanying uncertainties. Depending on the implementation, the predicted or transformed second digital marker state probability distribution can be combined with or compared to the current sensor data to update the model. The updated model can produce a prediction of the state variables, which is a third digital marker state probability distribution. Based on the third digital marker state probability distribution, the predicted marker position can be generated.

[0063] In an alternative embodiment, instead of using and comparing the probability distributions of each digital marker state in the process, the state can be used to represent a reasonable probability distribution of marker positions, where each marker position is a state variable that produces the predicted marker position.

[0064] The third digital marker state probability distribution can represent the probability of any particular marker state that is at least partially based on data ingested by the inference engine.

[0065] The generation operation 408 generates the predicted digit marker position based at least in part on the probability distribution of the third digit marker state. In one implementation, the generation operation 408 uses an inference engine to predict the marker position based on the most likely digit marker state. In another implementation, the inference engine can predict the marker position by determining the marker position probability distribution based on, for example, the probability distribution of the third digit marker state and predicting the digit marker position based on the marker position probability distribution.

[0066] The relationship between the predicted marker locations and digit marker states can be based, at least in part, on the geometric features of the digit marker. For example, the orientation and location of the digit marker can only represent a specific region on which the digit marker can make digit marks. An inference engine that restricts these potential digit marker states and evaluates the probability of each of these states can significantly restrict this region and generate highly probable locations for the markers.

[0067] The merging operation 410 combines data generated by the operating system and other software-based predictive models with data generated based on one or more of the probability distributions of the third digital marker state, the marker position probability distribution, and the sensor-based predicted marker position to produce a validated predicted marker position. These software-based predictive models may be independent of sensor data or may rely on older sensor data to determine reasonable predictions. These software-based predictive models can operate faster and / or more frequently and can provide predicted digital marker positions or data to be combined with sensor-based data. The software-based predicted position can be considered a “wet ink” marker, which will not ultimately be determined as a “dry ink” marker until more updated sensor-based data is combined or otherwise considered. Combining the sensor-based predicted position data with the software-based predicted position data produces a more robust predicted digital marker position. Merging operation 410 can combine predicted positions generated by different methods by averaging the coordinates of predicted markers generated by different models, taking the median of the coordinates, sorting and selecting the best predicted marker position, or using one or more consensus mechanisms to determine the predicted marker. The probability distribution of the third digital marker state can be used as the previous digital marker state probability in the next or later iteration. An implementation is envisioned where data from software-based methods is not merged to predict marker positions, and operation 400 omits operation 410.

[0068] An alternative implementation is envisioned, in which the digital marker state probability distribution is used to generate the marker position probability distribution before being merged or combined, allowing the inference engine to produce the marker position probability distribution at each step. In this implementation, the inference engine can determine a first marker position probability distribution using a first digital marker state probability distribution. The inference engine can determine a previous marker position probability distribution using a previous digital marker state probability distribution. The inference engine can combine the first marker position probability distribution and the previous marker position probability distribution to produce a second marker position probability distribution. The inference engine can acquire current digital marker sensor data and user interaction sensor data, and combine them to determine one or more of the current sensor digital marker state probability distribution and the current sensor marker position probability distribution. The inference engine can combine the second marker position probability distribution with one or more of the current sensor digital marker state probability distribution and the current sensor marker position probability distribution. These distributions can be combined using one or more of convolution, linear transformation, state machine, machine learning implementation, Kalman filter, and other inference methods. Based on this combination, the inference engine can produce a third marker position probability distribution. Based on the probability distribution of the third marker location, the inference engine can directly predict the marker location, or it can combine the predicted location or the probability distribution of the third marker location with other publicly disclosed software-based predictions to predict the marker location.

[0069] Figure 5 An example implementation of memory 500 is shown. Memory 500 can be an implementation of memory 606. User interaction determiner 502 can determine user interaction data based on user interaction with user interaction sensors. In various implementations, user interaction determiner 502 can perform any aspect of determination operation 302. Geometric feature determiner 504 can determine the geometric features of the digital marker. In various implementations, geometric feature determiner 504 can perform any aspect of determination operation 304.

[0070] Predictive marker generator 506 generates the predicted digit markers. Predictive marker generator 506 as a whole can perform aspects of one or more of operations 306, 400, 402, 404, 406, 408, and 410. First digit marker state probability generator (DMSPG) 512 generates a first digit marker state probability distribution, possibly based on user interaction data determined by user interaction determiner 502. In one implementation, first DMSPG 512 can perform aspects of operation 402.

[0071] The distribution combiner 560 combines the current digit marker state data derived from the current user interaction data with the previous digit marker state probability distribution data. The distribution combiner 560 may have a second DMSPG 514 to obtain the combined first digit marker state probability distribution and the previous digit marker state probability distribution, and to generate a second digit marker state probability distribution. In various implementations, the distribution combiner 560 and / or the second DMSPG 514 may perform aspects of operation 404.

[0072] The marker position predictor 562 generates the predicted marker position based on the second digital marker probability distribution and current sensor data. The marker position predictor 562 may transform the second digital marker probability distribution before combining it with the current sensor data. The marker position predictor 562 uses a third DMSPG 516 to combine the transformed second digital marker state probability distribution with the current sensor data (e.g., digital marker sensor data features and / or user interaction sensor data, possibly determined by one or more of the user interaction determiner 502 and the geometric feature determiner 504) and generates a third digital marker state probability distribution. Based on this third digital marker state probability distribution, the marker position predictor 562 predicts the marker position. In various implementations, the marker position predictor 562 may perform any aspect of one or more of operations 406, 408, and 410. In various implementations, the third DMSPG 616 may perform aspects of operation 406. As discussed in this paper, an embodiment is envisioned that uses a marker position probability distribution instead of a digital marker state probability distribution.

[0073] Figure 6 An example computing device 600 is illustrated for implementing the features and operations of the described technology. The computing device 600 may be a remotely controlled device or a physically controlled device, and is an example of a network-connected and / or network-capable device, and may be a client device such as a laptop computer, mobile device, desktop computer, tablet; server / cloud device; Internet of Things device; electronic accessory; or other electronic device. The computing device 600 includes one or more processors 602 and a memory 604. The memory 604 generally includes both volatile memory (e.g., RAM) and non-volatile memory (e.g., flash memory). The memory 604 may be an implementation of memory 500. An operating system 610 resides in memory 604 and is executed by the processors 602.

[0074] In example computing device 600, such as Figure 6As shown, one or more modules or components, such as application 650, state machine, inference method, inference engine, weighted algorithm, Kalman filter, frequency-weighted Kalman filter, extended Kalman filter, unscented Kalman filter, discriminant Kalman filter, adaptive Kalman filter, Kalman-Busch filter, hybrid Kalman filter, Kalman smoother, α-β filter, inverse variance weighting, covariance intersection, stochastic relation, state transition matrix, kernel adaptive filter, recursive least squares filter, particle filter estimator, moving time domain estimator, recursive Bayesian estimator, and predictor-corrector method, software-based predictor, operating system-based predictor, averager, median calculator, statistical method module, user interaction determiner, geometric feature determiner, predictive marker generator, first digital marker state probability generator, second digital marker state probability generator, third digital marker state probability generator, and capacitive sensor driver, are loaded into operating system 610 on memory 604 and / or memory 620 and executed by processor 602. The memory 620 may include one or more tangible storage medium devices and may store digital marker state probability distributions, digital marker states, digital marker state probabilities, digital marker state uncertainties, marker state data, geometric features, user interaction sensor data, user interaction patterns, digital marker data, digital markers, predicted marker positions, digital marker positions, digital marker orientations, digital marker speeds, prediction models, user postures, hand orientations, user's dominant hand, user characteristics, touch pattern data, user shadow data, normal distributions, measurement data, noise in the measurement data, estimated noise with uncertainties reflected in the probability distribution, verified predicted marker positions, sensor-based predicted marker positions, software-based predicted marker positions, marker position probability distributions, local and globally unique identifiers, requests, responses, and other data, and may be local to computing device 600 or may be remote and communicatively connected to computing device 600.

[0075] The computing device 600 includes a power supply 616, which is powered by one or more batteries or other power sources and provides electrical energy to other components of the computing device 600. The power supply 616 may also be connected to an external power source for overclocking or recharging the built-in battery or other power source.

[0076] Computing device 600 may include one or more communication transceivers 630, which may be connected to one or more antennas 632 to provide network connectivity (e.g., mobile phone networks) to one or more other server and / or client devices (e.g., mobile devices, desktop computers, or laptop computers). The computing device 600 may further include a network adapter 636, which is a type of computing device. The computing device 600 can use this adapter and any other type of computing device to establish connections on a wide area network (WAN) or local area network (LAN). It should be understood that the network connections shown are examples, and other computing devices and apparatuses can be used to establish communication links between the computing device 600 and other devices. The transceiver 630 and antenna 632 may also be configured to communicatively couple to and / or communicate with a digital marker, such as digital marker 106.

[0077] Computing device 600 may include one or more input devices 634 (e.g., a keyboard or mouse) that allow a user to type commands and information. These and other input devices may be coupled to computer system 400 via one or more interfaces 638, such as a serial port interface, a parallel port, or a universal serial bus (USB). Computing device 600 may also include a display 622, such as a touchscreen display. Display 622 may be an implementation of user input sensor 104, in one implementation of which it may be a capacitive touch sensor. Display 622 may be configured to receive sensor data from a digital marker (e.g., digital marker 106). Display 622 may also sense touches from the digital marker on display 622. In other implementations, the user input sensor may be a trackpad or other non-display trackpad that is communicatively coupled to a system bus to facilitate sensing communication with display 622 or the touchscreen display.

[0078] The computing device 600 may include a wide variety of tangible processor-readable storage media and intangible processor-readable communication signals. Tangible processor-readable storage can be embodied by any available medium accessible to the computing device 600, and includes both volatile and non-volatile storage media, and removable and non-removable storage media. Tangible processor-readable storage media does not include communication signals (e.g., the signals themselves), but includes volatile and non-volatile, removable and non-removable storage media implemented using any method or technology for storing information such as processor-readable instructions, data structures, program modules, or other data. Tangible processor-readable media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices, or any other tangible medium that can be used to store desired information and is accessible to the computing device 600. In contrast to tangible processor-readable storage media, intangible processor-readable communication signals can embody processor-readable instructions, data structures, program modules, or other data using modulated data signals such as carrier waves or other signal transmission mechanisms. The term "modulated data signal" refers to a signal in which one or more characteristics are set or altered by encoding information in the signal. As an example, and not a limitation, intangible communication signals include signals that propagate through wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media).

[0079] The various software components described herein can be executed by one or more processors, which may include logic machines configured to execute hardware or firmware instructions. For example, a processor may be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, transform the state of one or more components, achieve technical effects, or otherwise obtain desired results.

[0080] Various aspects of the processor and memory can be integrated into one or more hardware logic components. Such hardware logic components may include, for example, field-programmable gate arrays (FPGAs), application-specific integrated circuits (PASICs / ASICs), application-specific standard products (PSSPs / ASSPs), systems-on-a-chip (SoCs), and complex programmable logic devices (CPLDs).

[0081] The terms "module," "program," and "engine" can be used to describe an aspect of a remotely controlled device and / or a physically controlled device implemented to perform a specific function. It will be understood that different modules, programs, and / or engines can be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine can be instantiated from different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" are intended to encompass single or grouped executable files, data files, libraries, drivers, scripts, database records, etc.

[0082] It should be understood that, as used herein, "service" refers to an application that can execute on one or more user sessions. A service can be used with one or more system components, programs, and / or other services. In some implementations, a service can run on one or more server computing devices.

[0083] The logical operations constituting embodiments of the invention described herein may be referred to differently as operations, steps, objects, or modules. Furthermore, it should be understood that logical operations may be performed in any order, with operations added or omitted as needed, regardless of whether an operation is marked or identified as optional, unless otherwise expressly stated or the particular order is inherent to the language of the claims.

[0084] An example method is provided. This example method includes the following operations: determining a user interaction pattern with a digital marker, the user interaction pattern being at least partially based on user interaction with a capacitive sensor of a computing device; determining geometric features of the digital marker; and generating a predicted digital mark position of the digital marker on a digital medium of the computing device, based at least partially on the determined geometric features of the digital marker and the determined user interaction pattern.

[0085] Another example method according to any of the foregoing methods is provided, wherein determining the predicted digit marker position of the digit marker further includes: generating a first digit marker state probability distribution based at least in part on the determined user interaction pattern, wherein the predicted digit marker position is generated based at least in part on the first digit marker state probability distribution.

[0086] Another example method according to any of the foregoing methods is provided, wherein determining the predicted digit marker position further includes generating a second digit marker state probability distribution based at least in part on a first digit marker state probability distribution and a previous digit marker state probability distribution.

[0087] Another example method according to any of the foregoing methods is provided, wherein determining the predicted digital marker position further includes generating a current marker state probability distribution based at least in part on the determined geometric features of the digital marker, the determined user interaction pattern, and a second digital marker state probability distribution.

[0088] Another example method according to any of the foregoing methods is provided, wherein determining the user interaction pattern further includes inferring at least one of the user's dominant hand, the user's hand size, and the user's hand orientation based on data generated by a capacitive sensor.

[0089] Another example method according to any of the foregoing methods is provided, wherein determining the user interaction pattern with respect to the digital marker includes detecting one or more of a user touch pattern on a capacitive sensor and a user shadow on a capacitive sensor.

[0090] Another example method according to any of the foregoing methods is provided, wherein determining a user interaction pattern with respect to a digital marker further includes sensing a pattern of user marks over time using a capacitive sensor, and determining a user characteristic based at least in part on the pattern of user marks over time, the user interaction pattern being generated based on the user characteristic. An example computing device having a processor and a memory is provided. The processor is provided configured to perform operations stored in the memory. The computing device includes: a user interaction determiner operable to determine a user interaction pattern with respect to a digital marker, the user interaction pattern being based at least in part on user interaction with a capacitive sensor of the device; a geometric feature determiner operable to determine geometric features of the digital marker; and a predictive marker generator operable to generate a predicted digital marker position of the digital marker based at least in part on the determined geometric features of the digital marker and the determined user interaction pattern.

[0091] Another example computing device according to any of the aforementioned devices is provided, wherein the predictive marker generator includes a first digital marker state probability generator operable to generate a first digital marker state probability distribution based at least in part on a determined user interaction pattern, wherein the predictive marker generator also generates the predicted digital marker position based at least in part on the first digital marker marker state probability distribution.

[0092] Another example computing device based on any of the aforementioned devices is provided, wherein the predictive marker generator further includes a second digital marker state probability generator operable to generate a second digital marker state probability distribution based at least in part on a first digital marker state probability distribution and a previous digital marker state probability distribution.

[0093] Another example computing device based on any of the aforementioned devices is provided, wherein the predictive marker generator further includes a third digital marker state probability generator operable to generate a third marker state probability distribution based at least in part on the determined geometric features of the digital marker, the determined user interaction pattern, and the second digital marker state probability distribution.

[0094] Another example computing device based on any of the aforementioned devices is provided, wherein the user interaction determiner is also operable to infer at least one of the user's dominant hand, the user's hand size, and the user's hand orientation based on data generated by a capacitive sensor.

[0095] Another example computing device based on any of the aforementioned devices is provided, wherein the user interaction determiner is also operable to detect one or more of a user touch pattern on a capacitive sensor and a user shadow on a capacitive sensor using a capacitive sensor.

[0096] Another example computing device according to any of the foregoing devices is provided, wherein the user interaction determiner is also operable to sense the pattern of user marks over time using a capacitive sensor, and to determine user characteristics based at least in part on the pattern of user marks over time, the user interaction pattern being generated based on the user characteristics.

[0097] One or more tangible example processor-readable storage media devices are provided for encoding processor-executable instructions for performing processes on an electronic computing device. The process includes: determining a user interaction pattern with respect to a digital marker, the user interaction pattern being at least partially based on user interaction with a capacitive sensor of the computing device; determining geometric features of the digital marker; and generating a predicted digital mark position of the digital marker on a digital medium of the computing device based at least partially on the determined geometric features of the digital marker and the determined user interaction pattern.

[0098] One or more other examples of tangible processor-readable storage medium devices according to any of the foregoing media are provided, wherein determining the predicted digital marker position of a digital marker further includes generating a first digital marker state probability distribution based at least in part on a determined user interaction pattern, wherein the predicted digital marker position is generated based at least in part on the first digital marker state probability distribution.

[0099] One or more other examples of tangible processor-readable storage medium devices based on any of the aforementioned media are provided, wherein determining the predicted digital marker position further includes generating a second digital marker state probability distribution based at least in part on a first digital marker state probability distribution and a previous digital marker state probability distribution.

[0100] A tangible processor-readable storage medium device is provided based on one or more other examples of any of the aforementioned media, wherein determining the predicted digital marker location further includes generating a current marker state probability distribution based at least in part on the determined geometric features of the digital marker, the determined user interaction pattern, and a second digital marker state probability distribution.

[0101] One or more other examples of tangible processor-readable storage medium devices based on any of the foregoing media are provided, wherein determining the user interaction mode further includes inferentially determining at least one of the user's dominant hand, the user's hand size, and the user's hand orientation based on data generated by a capacitive sensor.

[0102] One or more other examples of tangible processor-readable storage medium devices according to any of the foregoing media are provided, wherein determining a user interaction pattern with respect to a digital marker includes detecting one or more of a user touch pattern on a capacitive sensor and a user shadow on a capacitive sensor.

[0103] An example system is provided, comprising: means for determining a user interaction pattern with respect to a digital marker, the user interaction pattern being at least partially based on user interaction with a capacitive sensor of a computing device; means for determining geometric features of the digital marker; and means for generating a predicted digital mark position of the digital marker on a digital medium of the computing device based at least partially on the determined geometric features of the digital marker and the determined user interaction pattern.

[0104] Another example system according to any of the foregoing systems is provided, wherein the means for determining the predicted digital marker position of the digital marker pen further includes means for generating a first digital marker pen state probability distribution based at least in part on the determined user interaction pattern, wherein the predicted digital marker position is generated at least in part based on the first digital marker pen state probability distribution.

[0105] Another example system according to any of the foregoing systems is provided, wherein the means for determining the predicted position of the digital marker further includes means for generating a second digital marker state probability distribution based at least in part on a first digital marker state probability distribution and a previous digital marker state probability distribution.

[0106] Another example system according to any of the foregoing systems is provided, wherein the means for determining the predicted digital marker position further includes means for generating a current marker state probability distribution based at least in part on the determined geometric features of the digital marker, the determined user interaction pattern, and the second digital marker state probability distribution.

[0107] Another example system according to any of the foregoing systems is provided, wherein the means for determining the user interaction pattern further includes means for inferentially determining at least one of the user's dominant hand, the user's hand size, and the user's hand orientation based on data generated by the capacitive sensor.

[0108] Another example system according to any of the foregoing systems is provided, wherein the means for determining a user interaction pattern with respect to a digital marker includes means for detecting one or more of a user touch pattern on a capacitive sensor and a user shadow on a capacitive sensor.

[0109] Another example system according to any of the foregoing systems is provided, wherein the means for determining a user interaction pattern with respect to a digital marker further includes means for sensing the pattern of the user mark over time with a capacitive sensor, and means for determining a user characteristic based at least in part on the pattern of the user mark over time, the user interaction pattern being generated based on the user characteristic.

[0110] While this specification contains numerous specific implementation details, these should not be construed as limiting any invention or potentially claimed scope, but rather as descriptions of features characteristic of a particular implementation of the specifically described technique. Some features described in this specification in the context of independent implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually in multiple implementations or in any suitable sub-combination. Furthermore, although features may be described above as functioning in certain combinations, and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0111] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order, or that all illustrated operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system components in the above embodiments should not be construed as requiring such separation in all embodiments, but rather it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0112] Therefore, a specific implementation of the subject matter has been described. Other embodiments are within the scope of the following claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes described in the drawings do not necessarily require the specific order or sequence shown to obtain the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0113] Several implementations of the described technology have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the cited claims.

Claims

1. A method comprising: Determine user interaction patterns for digital markers, said user interaction patterns being at least in part based on user interaction with capacitive sensors of computing devices; Determine the geometric features of the digital marker; The first digital marker state probability distribution is generated based at least in part on the determined user interaction pattern; as well as The predicted digital marker position of the digital marker on the digital medium of the computing device is generated based at least in part on the determined geometric features of the digital marker, the probability distribution of the state of the first digital marker, and the determined user interaction pattern.

2. The method according to claim 1, characterized in that, Determining the predicted location of the digital marker also includes: The second digital marker state probability distribution is generated at least in part based on the first digital marker state probability distribution and the previous digital marker state probability distribution.

3. The method according to claim 2, characterized in that, Determining the predicted location of the digital marker also includes: The current marker state probability distribution is generated based at least in part on the determined geometric features of the digital marker, the determined user interaction pattern, and the second digital marker state probability distribution.

4. The method according to claim 1, characterized in that, Determining the user interaction mode further includes inferring at least one of the user's dominant hand, user's hand size, and user's hand orientation based on data generated by the capacitive sensor.

5. The method according to claim 1, characterized in that, Determining the user interaction pattern with respect to the digital marker includes using the capacitive sensor to detect one or more of the user touch pattern on the capacitive sensor and the user shadow on the capacitive sensor.

6. The method according to claim 1, characterized in that, Determining the user interaction patterns for digital markers also includes: The capacitive sensor is used to sense the pattern of user markings over time; and User characteristics are determined at least in part based on the patterns of user tags over time, and the user interaction patterns are generated based on the user characteristics.

7. A computing device having a processor and a memory, the processor being configured to perform operations stored in the memory, the computing device comprising: User interaction determiner, operable to determine user interaction patterns with respect to a digital marker, the user interaction patterns being at least in part based on user interaction with capacitive sensors of the device; A geometric feature determiner, operable to determine the geometric features of the digital marker; A first digital marker state probability generator, which is operable to generate a first digital marker state probability distribution based at least in part on a determined user interaction pattern. as well as A predictive marker generator is operable to generate predicted digital marker positions for the digital marker based at least in part on determined geometric features of the digital marker, a probability distribution of the first digital marker state, and a determined user interaction pattern.

8. The computing device according to claim 7, characterized in that, The prediction tag generator also includes: A second digital marker state probability generator is operable to generate a second digital marker state probability distribution based at least in part on the first digital marker state probability distribution and a previous digital marker state probability distribution.

9. The computing device according to claim 8, characterized in that, The prediction tag generator also includes: A third digital marker state probability generator is operable to generate a third marker state probability distribution based at least in part on the determined geometric features of the digital marker, the determined user interaction pattern, and the second digital marker state probability distribution.

10. The computing device according to claim 7, characterized in that, The user interaction determiner can also be operated to infer at least one of the user's dominant hand, user's hand size, and user's hand orientation based on data generated by the capacitive sensor.

11. The computing device according to claim 7, characterized in that, The user interaction determiner can also be operated to detect one or more of the user touch pattern on the capacitive sensor and the user shadow on the capacitive sensor using the capacitive sensor.

12. The computing device according to claim 7, characterized in that, The user interaction determiner can also operate as follows: The capacitive sensor is used to sense the pattern of user markings over time; as well as User characteristics are determined at least in part based on the patterns of user tags over time, and the user interaction patterns are generated based on the user characteristics.

13. One or more tangible processor-readable storage media devices, said one or more tangible processor-readable storage media devices encoding processor-executable instructions for performing a process on an electronic computing device, said process comprising: Determine user interaction patterns for digital markers, said user interaction patterns being at least in part based on user interaction with capacitive sensors of computing devices; Determine the geometric features of the digital marker; The first digital marker state probability distribution is generated based at least in part on the determined user interaction pattern; as well as The predicted digital marker position of the digital marker on the digital medium of the computing device is generated based at least in part on the determined geometric features of the digital marker, the probability distribution of the state of the first digital marker, and the determined user interaction pattern.

14. The one or more tangible processor-readable storage media devices according to claim 13, characterized in that, Determining the predicted location of the digital marker also includes: The second digital marker state probability distribution is generated at least in part based on the first digital marker state probability distribution and the previous digital marker state probability distribution.

15. The one or more tangible processor-readable storage media devices according to claim 14, characterized in that, Determining the predicted location of the digital marker also includes: The current marker state probability distribution is generated based at least in part on the determined geometric features of the digital marker, the determined user interaction pattern, and the second digital marker state probability distribution.

16. The one or more tangible processor-readable storage media devices according to claim 15, characterized in that, Determining the user interaction mode further includes inferring at least one of the user's dominant hand, user's hand size, and user's hand orientation based on data generated by the capacitive sensor.

17. The one or more tangible processor-readable storage media devices according to claim 16, characterized in that, Determining the user interaction pattern with respect to the digital marker includes using the capacitive sensor to detect one or more of the user touch pattern on the capacitive sensor and the user shadow on the capacitive sensor.

Images