Writing track prediction method, device and equipment

Abstract

The invention discloses a writing track prediction method, device and equipment, and the method comprises the steps: obtaining a plurality of pieces of report point information of a touch object which is touching touch display equipment, and the report point information comprises report point coordinates and report point time; processing the report point information in the current sliding window to obtain a current report point sequence; inputting the current report point sequence into a prediction model, and predicting at least one future report point coordinate through the prediction model; the prediction model is obtained by training a deep learning-based transformer model by using training data, and the prediction model is used for generating the at least one future report point coordinate according to the current report point sequence; and performing track rendering on the last report point coordinate and at least one future report point coordinate. The writing track prediction method, device and equipment provided by the invention have the advantage of high prediction accuracy.

Description

Technical Field

[0001] This application relates to the field of writing technology, and in particular to a writing trajectory prediction method, apparatus, and device. Background Technology

[0002] Styluses are currently the most common accessory for tablets, frequently used for note-taking and drawing in office settings. Due to the time required for touch input, point marking, and drawing, writing on a tablet with a stylus has a noticeable lag compared to writing on paper with a regular pen.

[0003] In related technologies, kinematic models are typically built using kinematic principles to predict subsequent writing trajectories. However, the prediction results of kinematic models are prone to bias, affecting user experience. Summary of the Invention

[0004] The purpose of this application is to provide a writing trajectory prediction method, apparatus, and device, which can solve the problem that the prediction results of related writing trajectory prediction technologies are biased and affect user use.

[0005] To achieve the above objectives, one aspect of this application provides a writing trajectory prediction method for a touch display device. The method includes: acquiring multiple reporting point information of a touch object currently touching the touch display device, the reporting point information including corresponding reporting point coordinates and reporting time; processing the reporting point information in the current sliding window to obtain a current reporting point sequence; inputting the current reporting point sequence into a prediction model, and predicting at least one future reporting point coordinate through the prediction model; the prediction model is trained using training data on a deep learning-based transformer model, and the prediction model is used to generate the at least one future reporting point coordinate based on the current reporting point sequence; and rendering the trajectory of the last reporting point coordinate and the at least one future reporting point coordinate; wherein the last reporting point coordinate is the reporting point coordinate corresponding to the latest reporting time among the multiple reporting point information.

[0006] To achieve the above objectives, another aspect of this application provides a writing trajectory prediction device for a touch display device. The device includes: an acquisition module for acquiring multiple reporting points of a touch object currently touching the touch display device, the reporting points including reporting point coordinates and reporting time; a processing module for processing the reporting point information in the current sliding window to obtain a current reporting point sequence; a learning module for inputting the current reporting point sequence into a prediction model and predicting at least one future reporting point coordinate through the prediction model; the prediction model is trained using training data on a deep learning-based transformer model, and the prediction model is used to generate the at least one future reporting point coordinate based on the current reporting point sequence; and a rendering module for rendering the trajectory of the last reporting point coordinate and the at least one future reporting point coordinate.

[0007] To achieve the above objectives, another aspect of the embodiments of this application provides a touch display device, including a memory and at least one processor; wherein the memory is used to store one or more computer instructions, which are executed by the processor to implement the method described above.

[0008] To achieve the above objectives, another aspect of the embodiments of this application provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the method described above.

[0009] In summary, the writing trajectory prediction method, apparatus, and device provided in this application, by employing the deep learning model Transformer to predict the coordinates of future points, can couple the coordinate information of the X-axis and Y-axis directions, learn the correlation between the X-axis and Y-axis directions, and better learn the motion trajectory rules, thereby improving the prediction accuracy. Attached Figure Description

[0010] Figure 1 A flowchart illustrating a writing trajectory prediction method provided in this application embodiment;

[0011] Figure 2 A schematic diagram of a writing trajectory prediction device provided in an embodiment of this application;

[0012] Figure 3 This is a structural diagram of a touch display device provided in an embodiment of this application. Detailed Implementation

[0013] As described in the background section, the kinematic models of related technologies suffer from biased predictions, affecting user experience. The inventors' research revealed the following reasons for this problem:

[0014] When an object is written on a touch display device, it generates multiple reporting point coordinates. Since the writing trajectory displayed on the touch display device is a planar line segment, the reporting point coordinates can be represented by the corresponding values ​​on the X-axis and Y-axis. Therefore, multiple reporting point coordinates include multiple X-axis values ​​and multiple Y-axis values. When using a kinematic model for prediction, multiple X-axis values ​​from the historical period are used to decouple and predict future X-axis values, and multiple Y-axis values ​​from the historical period are used to decouple and predict future Y-axis values. In other words, the kinematic model prediction method decouples the X and Y directions separately, ignoring the correlation between the X and Y directions, leading to prediction bias.

[0015] Based on this, embodiments of this application consider a prediction method that can correlate the X and Y directions. The transformer model can capture the dependencies between different positions in different input sequences through an attention mechanism; therefore, the inventors of this application conceived of using the transformer model to predict future trajectories.

[0016] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments.

[0017] Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0018] This application provides a writing trajectory prediction method, which can be applied to touch display devices. The touch display device can be a computer, tablet, laptop, smartphone, or other display device with a touch surface. The touch display device may have writing software installed, allowing users to write by swiping their fingers or styluses on the software interface. The touch display device draws the detected touch trajectory and displays it on the screen as a writing trajectory.

[0019] Figure 1 This is a schematic diagram of the writing trajectory prediction method provided in the embodiments of this application, with reference to... Figure 1 The writing trajectory prediction method provided in this application embodiment may include the following steps:

[0020] S101. Obtain the reporting information of the object being touched on the touch display device.

[0021] Specifically, the reporting information includes the corresponding reporting coordinates and the reporting time. In other words, the reporting information includes the reporting time and the reporting coordinates obtained at that time. Touch display devices typically collect and report the reporting coordinates (x, y) of the touched object at preset time intervals (e.g., 10ms). When the touched object slides on the touch screen of the touch display device, the touch device will acquire multiple reporting coordinates.

[0022] Due to the existence of errors, there may be error points and redundant points among multiple reported coordinates. Based on this, alternatively, after obtaining the reported coordinates, abnormal reported coordinates can be removed and missing reported coordinates can be added. Smoothing methods such as curve fitting can also be used to smooth the reported coordinates by referring to the historical reported coordinates before them, in order to improve the accuracy of the prediction.

[0023] S102. Process the reporting information within a sliding window to obtain the current reporting sequence.

[0024] Specifically, a sliding window can be used to divide the multiple reporting points acquired by the device. The window length of the sliding window is n, which represents that there are n consecutive reporting points in the sliding window. If the current time is T, the current sliding window can be formed by the reporting points acquired by the device at time T and the N-1 reporting points acquired by the device before time T.

[0025] For example, a sliding window of fixed length n can be set starting from the first reported point information obtained by the device. The sliding window slides forward one position at a time, and the partitioning ends when the sliding window reaches the reported point information obtained by the device at the current time. For instance, if the device obtains multiple reported point information {P1, P2, ..., Pn, Pn+1, ..., P2n}, and the sliding window step size is n, and the first reported point information obtained by the device is P1, and the currently obtained reported point information is P2n, then the first sliding window is {P1, P2, ..., Pn}, the second sliding window is {P2, P3, ..., Pn, Pn+1}, and the current sliding window is {Pn+1, ..., P2n}.

[0026] It should be noted that when the object is first slid across the device's touchscreen, the reporting frequency remains constant, but the movement speed is relatively slow with no noticeable delay. Simultaneously, the minimum window length for the sliding window must be met.

[0027] Therefore, when the total number of reported points is less than M, trajectory prediction is not performed, and the data is displayed normally. Trajectory prediction is performed when the total number of reported points meets M. M can be preset. For example, M can be 10, 30, etc., and this application does not limit this setting. It is understandable that since the device can collect approximately 300 reported point information per second, for example, 30 reported points can be collected in about 100 milliseconds. If N is 10, it can be completed in just over 30 milliseconds. This is a very short time for the user, even imperceptible, and will not cause a significant delay.

[0028] When an object is touched and slids on the touchscreen of a touch display device, the device acquires multiple point coordinates. These point coordinates are discrete points, therefore they need to be rendered as a curve. This curve can be viewed as the trajectory of the touched object. Since the trajectory of the touched object changes over time, the N point coordinates in a sliding window also change over time. A point coordinate can be formed by X-axis and Y-axis coordinates; therefore, the N point coordinates can be broken down into N X-axis coordinates and N Y-axis coordinates. Furthermore, the N X-coordinates and N Y-coordinates change over time.

[0029] Since the device acquires the coordinates of the reported points at a fixed scanning frequency, the time interval between two consecutive reported point coordinates is a constant. Therefore, the velocity V1 can be obtained by using the ratio of the coordinate difference ΔX between two consecutive X-axis coordinates to the time interval; the acceleration a1 can be obtained by using the ratio of the velocity difference ΔV1 between two consecutive X-axis coordinates to the time interval. Similarly, the velocity V2 can be obtained by using the ratio of the coordinate difference ΔY between two consecutive Y-axis coordinates to the time interval; the acceleration a2 can be obtained by using the ratio of the velocity difference ΔV2 between two consecutive Y-axis coordinates to the time interval. The slope of the line connecting the two reported point coordinates can be obtained by using the ratio of ΔY to ΔX, or the inclination angle of the line connecting the two reported point coordinates relative to the X-axis can be obtained by using the slope. Both the slope and the inclination angle can be referred to as the inclination value.

[0030] The coordinates of all reporting points within a sliding window can be concatenated to form a reporting sequence, a [N, 2] matrix; alternatively, the coordinates and velocities of all reporting points within a sliding window can be concatenated to form a [N, 4] matrix; alternatively, the coordinates, velocities, and accelerations of all reporting points within a sliding window can be concatenated to form a [N, 6] matrix; or alternatively, the coordinates, velocities, accelerations, and tilt values ​​of all reporting points within a sliding window can be concatenated to form a [N, 7] matrix. The coordinates, velocities, accelerations, and tilt values ​​of the reporting points are collectively referred to as feature elements. The more feature elements a reporting point sequence contains, the easier it is to better mine and utilize related features, and more information can be coupled in the future, which is beneficial for the learning and prediction of the prediction model.

[0031] For example, the current reporting sequence can be formed by concatenating the reporting coordinates, reporting velocity, and reporting acceleration of all reporting points in the current sliding window; or, the current reporting sequence can be formed by concatenating the reporting coordinates, reporting velocity, reporting acceleration, and tilt value of all reporting points in the current sliding window.

[0032] S103. Input the current reporting point sequence into the prediction model, and predict at least one future reporting point coordinate through the prediction model.

[0033] The prediction model can be trained using training data on a deep learning-based transformer model, and it can generate at least one future reporting point coordinate based on the current reporting point sequence. Specifically, the training data for training the transformer model can be obtained from a historical sliding window, which can be formed before the current sliding window. The historical reporting point sequence formed by the reporting point information in the historical sliding window can be used as training features, and the next or a historical coordinate point of the historical sliding window can be used as the point to be predicted, forming a training sample.

[0034] For example, a device acquires multiple reporting points {P1, P2, ..., Pn, Pn+1, ..., P2n}, with a sliding window step size of n, and predicts three future coordinates. For instance, using a historical sliding window {Pn-2, Pn-1, ..., P2n-3}, the reporting sequence of that historical sliding window is obtained. This historical reporting sequence is used as training features, and P2n-2, P2n-1, and P2n are used as points to be predicted, forming a training sample. Similarly, using a historical sliding window {Pn-3, Pn-2, ..., P2n-4}, the reporting sequence of that historical sliding window is obtained. This historical reporting sequence is used as training features, and P2n-3, P2n-2, and P2n-1 are used as points to be predicted, forming a training sample.

[0035] This application embodiment can employ a deep learning Transformer model to learn future reporting information based on historical reporting information. The reporting sequence is used as input to the encoder, and initial self-learning parameters are used as input to the decoder. An attention mechanism is applied by the encoder to extract effective prior feature information and the decoder's features, and future coordinates are predicted using historical coordinates and similarity.

[0036] Specifically, the Transformer model may include an encoder, a decoder, and a transformer. Step S103 above may include the following:

[0037] S1031. Input the current reporting point sequence into the encoder of the prediction model, and use the encoder of the prediction model to extract features to obtain the historical trajectory feature sequence; S1032. Input the random self-learning sequence into the decoder of the prediction model, and use the decoder of the prediction model to extract features to obtain the self-learning feature sequence, and use the historical trajectory feature sequence to guide the self-learning feature sequence to generate the predicted future trajectory sequence; S1033. Convert the predicted future trajectory sequence into K future reporting point coordinates.

[0038] Specifically, in step S1031, the encoder may include a feedforward neural network and a self-attention mechanism. The feedforward neural network is responsible for extracting features from the input reporting sequence to capture its semantic information. The feedforward neural network can be used to perform a non-linear transformation on the representation of each position. This transformation helps the model better capture complex patterns and features in the input sequence, highlighting the weights of important time points and weakening unimportant information. The self-attention mechanism can be used to interact with the feature information at different positions in the reporting sequence, calculate the correlation between positions, correlate the reporting coordinate information at each position, and capture the dependencies and trajectory shape features between the feature information.

[0039] Step S1031 may include using a feedforward neural network to perform a nonlinear transformation on the current reporting sequence to obtain a first transformed sequence. For example, matrix transformation may be used to transform the [N, 6] matrix mentioned above into an [N, 256] matrix, and then a nonlinear function may be used to simplify the [N, 256] matrix for easier subsequent calculations.

[0040] Step S1031 may further include using a self-attention mechanism to calculate the correlation between each element of the first transformation sequence and all elements of the first transformation sequence to obtain a historical trajectory feature sequence. Compared to kinematic model prediction, it has the advantage of associating the X-axis coordinate with the Y-axis coordinate, which helps the model predict more accurately.

[0041] In step S1032, the random learning sequence can be obtained by concatenating K random coordinates to form a [K, 2] matrix. The value of K is related to the number of future coordinates predicted by the prediction model. If the prediction model predicts two future coordinates, then K = 2, and the two random coordinates are concatenated to form a [2, 2] matrix of random self-learning sequence.

[0042] Additionally, the decoder may include a feedforward neural network, a self-attention mechanism, and a cross-attention mechanism. Step S1032 may include: using a feedforward neural network to perform a nonlinear transformation on the random self-learning sequence to obtain a second transformed sequence; using a self-attention mechanism to calculate the correlation between each element of the second transformed sequence and all elements of the second transformed sequence to obtain a self-learning feature sequence; and using a cross-attention mechanism to perform cross-attention and fusion between the self-learning feature sequence and the historical trajectory feature sequence to obtain a predicted future trajectory sequence.

[0043] The feedforward neural network and self-attention mechanism function in the same way as in the encoder. The cross-attention mechanism is used by the decoder to focus on the feature information of different positions in the encoder output when generating the output at each position, and to use the encoder features to guide the generation of the predicted future trajectory sequence.

[0044] In step S1033, a fully connected neural network can be used to transform the decoder's predicted future trajectory sequence into K future reporting point coordinates.

[0045] As mentioned above, the prediction model can be trained using training data obtained from historical sliding windows on a deep learning-based transformer model. The historical reporting sequence formed by the reporting information in the historical sliding window is used as training features, and the next or a historical coordinate point of that historical sliding window is used as the point to be predicted, forming a training sample.

[0046] During model training, the reported point sequence formed by the historical sliding window can be fed into the encoder, decoder, and fully connected neural network to obtain K predicted coordinates. The loss value can be determined based on these K predicted coordinates and the points to be predicted in the training sample. The cross-entropy function can be used as the loss function. After determining the loss value, the generative model can be trained based on it. Convergence conditions can be preset. After extensive training, when the preset convergence conditions are met, the prediction model is considered complete, and the prediction model is obtained. The convergence condition can be that the loss value reaches a stable state.

[0047] In the field of handwriting, high prediction accuracy is required. Predicted trajectory accuracy can be divided into tangent direction accuracy and normal direction accuracy. Tangent direction accuracy can be considered as the accuracy of following distance, while normal direction accuracy can be considered as the accuracy of deviation distance. Deviation distance accuracy has higher requirements than following distance accuracy. This is because if normal direction accuracy is poor, the predicted trajectory is prone to "flying lines," severely impacting the user experience. Therefore, the prediction model can assign a larger optimization weight to normal direction accuracy.

[0048] The writing trajectory prediction method provided in this application embodiment may further include S104, rendering the trajectory of the last reporting point coordinates and at least one future reporting point coordinates.

[0049] Specifically, the coordinates of the last reported point are the coordinates of the reported point corresponding to the latest reported time among multiple reported point information. For example, multiple reported point information include the reported point coordinates (x1, y1) at time T1, (x2, y2) at time T2, (x3, y3) at time T3, ..., and the reported point coordinates (xn, yn) at time Tn. Among them, time Tn is the latest reported time, so (xn, yn) is the coordinate of the last reported point.

[0050] Furthermore, when an object is swiped across the device's touchscreen, the reported points received by the device are discrete. Therefore, these discrete reported points need to be rendered as curves. For example, after determining multiple subsequent predicted points of the received points, a writing trajectory is further rendered based on these predicted points and / or the received points, and these writing trajectories are drawn and displayed on the display position of the touchscreen device.

[0051] In this embodiment, the number of rendered writing trajectories is consistent with the number of predicted points, and the writing trajectory lies between received points and predicted points, or between pairs of predicted points. When the device activates the writing software, it indicates that the device has entered writing mode. Two canvas layers can be created in the writing software: a vector line layer and a non-vector line layer. Layer A is the vector line layer, and layer B is the non-vector line layer, with layer A placed above layer B. The display position refers to the location of the vector line layer.

[0052] As an optional implementation, layer A is a prediction display layer, and layer B is a true trajectory layer. The prediction display layer displays the predicted writing trajectory calculated by the prediction algorithm, while the true trajectory layer displays the actual writing trajectory generated by the actual touch points. For example, when a true touch point is detected, each true touch point is connected to the next received true touch point to generate a true writing trajectory, which is displayed in the true trajectory layer. The predicted writing trajectory generated by the prediction method is displayed in the prediction display layer, and there is no difference in the state of the writing trajectories displayed on the screen between the true trajectory layer and the prediction display layer, thus avoiding visual differences. Furthermore, after the predicted writing trajectory is displayed in the prediction display layer, if a true touch point is received and the true writing trajectory in the true trajectory layer is updated, the true writing trajectory will be synchronously updated in the prediction display layer. This allows for prediction based on the actual touch points when predicting the next writing trajectory, resulting in a more accurate predicted writing trajectory.

[0053] As another optional implementation, after displaying the predicted writing trajectory at the display position, it monitors whether a real writing action corresponding to the predicted writing trajectory is received. If a real writing action is received, the predicted writing trajectory is updated according to the real writing action; if no real writing action is received, it indicates that the current writing action has ended, and the predicted writing trajectory is cleared. For example, there is a relatively long time difference between every two writing actions. During the writing process, the device checks for new touch points at fixed time intervals. If no new touch point is received within a preset time, it indicates that the current writing action has ended. If a predicted writing trajectory has already been generated, it is then cleared.

[0054] As another optional implementation, after displaying the predicted writing trajectory at the display position, the system waits to receive the actual touch point corresponding to the current predicted writing trajectory before performing the next writing trajectory prediction. For example, after displaying the predicted writing trajectory on the prediction display layer, the system waits to receive the actual touch point. Upon receiving the actual touch point, the actual trajectory is updated to both the actual trajectory layer and the prediction display layer. Then, the next writing trajectory prediction begins, calculating the writing speed and direction based on the existing touch points, and predicting the position of the next touch point. Because the system waits to receive the actual touch point before performing the next writing trajectory prediction, each predicted writing trajectory is generated based on the actual touch point, making the predicted writing trajectory closer to the actual trajectory.

[0055] It should be noted that if the input to the prediction model is a floating-point number, and pixel-level units are used during line segment rendering, the precision deviation of the decimal part is small and imperceptible to the human eye in actual writing. Therefore, rounding can be used to convert it to an integer. Specifically, before inputting the current reporting sequence into the prediction model, the process can also include rounding each element in the current reporting sequence to form a current reporting sequence where each element is an integer.

[0056] refer to Figure 2 This application also provides a writing trajectory prediction device, which can be used in a touch display device. The device may include an acquisition module 201, a processing module 202, a learning module 203, and a rendering module 204. The acquisition module 201 is used to acquire multiple reporting points of a touch object currently touching the touch display device. The reporting point information may include reporting point coordinates and reporting time. The processing module 202 is used to process the reporting point information in the current sliding window to obtain the current reporting point sequence. The learning module 203 is used to input the current reporting point sequence into a prediction model and predict at least one future reporting point coordinate. The prediction model is trained using training data on a deep learning-based transformer model. The rendering module 204 is used to render the trajectory of the last reporting point coordinate and at least one future reporting point coordinate.

[0057] Optionally, the learning module 203 can input the current reporting sequence into the encoder of the prediction model and use the encoder of the prediction model to extract features to obtain the historical trajectory feature sequence. Specifically, the learning module 203 can use a feedforward neural network to perform a nonlinear transformation on the current reporting sequence to obtain a first transformation sequence; the learning module 203 can also use a self-attention mechanism to calculate the correlation between each element of the first transformation sequence and all elements of the first transformation sequence to obtain the historical trajectory feature sequence.

[0058] The learning module 203 can also input the random self-learning sequence into the decoder of the prediction model, and use the decoder to extract features to obtain a self-learning feature sequence. It can also use historical trajectory feature sequences to guide the self-learning feature sequence to generate a predicted future trajectory sequence. Specifically, the learning module 203 can use a feedforward neural network to perform a nonlinear transformation on the random self-learning sequence to obtain a second transformation sequence. It can also use a self-attention mechanism to calculate the correlation between each element of the second transformation sequence and all elements of the second transformation sequence to obtain a self-learning feature sequence. Furthermore, it can use a cross-attention mechanism to perform cross-attention and fusion between the self-learning feature sequence and the historical trajectory feature sequence to obtain a predicted future trajectory sequence.

[0059] Learning module 203 can also use a fully connected neural network to transform the predicted future trajectory sequence into K future point coordinates.

[0060] Optionally, the processing module 202 can round each element in the current reporting sequence to form a current reporting sequence in which each element is an integer.

[0061] refer to Figure 3This application also provides a touch display device that integrates the writing trajectory prediction device provided in this application. The touch display device may include an input device 303, an output device 304, a memory 302, and one or more processors 301. The memory 302 may store one or more programs; when one or more programs are executed by one or more processors 301, the one or more processors 301 implement the writing trajectory prediction method provided in the above embodiments. The input device 303, output device 304, memory 302, and processor 301 can be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.

[0062] The memory 302 serves as a computing device-readable storage medium, capable of storing software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the writing trajectory prediction method in any embodiment of this application (e.g., handwriting detection module 31, handwriting prediction module 32, and line drawing module in the writing trajectory prediction device). The memory 302 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the device, etc. Furthermore, the memory 302 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 302 may further include memory remotely located relative to the processor 301, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0063] Input device 303 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 304 may include display devices such as a display screen.

[0064] The processor 301 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 302, thereby realizing the writing trajectory prediction method described above.

[0065] The writing trajectory prediction device, equipment, and computer provided above can be used to execute the writing trajectory prediction method provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0066] This application embodiment also provides a storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are used to perform the writing trajectory prediction method provided in the above embodiments. The writing trajectory prediction method includes: acquiring multiple reporting point information of a touch object that is touching a touch display device, the reporting point information including reporting point coordinates and reporting time; processing the reporting point information in the current sliding window to obtain a current reporting point sequence; inputting the current reporting point sequence into a prediction model, and predicting at least one future reporting point coordinate through the prediction model; the prediction model is obtained by training a deep learning-based transformer model using training data; and rendering the trajectory of the last reporting point coordinate and at least one future reporting point coordinate.

[0067] Storage medium – any type of memory device or storage device. The term “storage medium” is intended to include: mounting media, such as CD-ROMs, floppy disks, or magnetic tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (e.g., hard disks or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. Furthermore, storage media may reside in a first computer system in which a program is executed, or may reside in a different second computer system connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term “storage medium” can include two or more storage media that may reside in different locations (e.g., in different computer systems connected via a network). Storage media may store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.

[0068] Of course, the computer-executable instructions provided in the embodiments of this application are not limited to the writing trajectory prediction method described above, but can also execute related operations in the writing trajectory prediction method provided in any embodiment of this application.

[0069] The writing trajectory prediction device, equipment, and storage medium provided in the above embodiments can execute the writing trajectory prediction method provided in any embodiment of this application. For technical details not described in detail in the above embodiments, please refer to the writing trajectory prediction method provided in any embodiment of this application.

[0070] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the protection scope of this application.

Claims

1. A writing trajectory prediction method for a touch display device, characterized in that, The method includes: Acquire multiple reporting points of the object being touched on the touch display device, the reporting points including corresponding reporting point coordinates and reporting time; The current reporting sequence is obtained by processing the reporting information in the current sliding window; The current reporting sequence is input into the prediction model, and at least one future reporting coordinate is predicted by the prediction model; the prediction model is trained on a deep learning-based transformer model using training data, and the prediction model is used to generate the at least one future reporting coordinate based on the current reporting sequence. The coordinates of the last reported point are used to render a trajectory with the coordinates of at least one future reported point; wherein, the coordinates of the last reported point are the coordinates of the reported point corresponding to the latest reported point time among the multiple reported point information.

2. The method according to claim 1, characterized in that, The step of inputting the current reporting sequence into the prediction model and predicting at least one future reporting coordinate through the prediction model includes: The current reporting sequence is input into the encoder of the prediction model, and the encoder of the prediction model is used to extract features to obtain the historical trajectory feature sequence. The random self-learning sequence is input into the decoder of the prediction model, and the decoder of the prediction model is used to extract features to obtain the self-learning feature sequence. The historical trajectory feature sequence is used to guide the self-learning feature sequence to generate a predicted future trajectory sequence. The predicted future trajectory sequence is transformed into K future point coordinates.

3. The method according to claim 2, characterized in that, The step of inputting the current reporting sequence into the encoder of the prediction model and using the encoder of the prediction model to extract features to obtain the historical trajectory feature sequence includes: A first transformed sequence is obtained by performing a nonlinear transformation on the current reporting sequence using a feedforward neural network. The self-attention mechanism is used to calculate the correlation between each element of the first transformation sequence and all elements of the first transformation sequence to obtain the historical trajectory feature sequence.

4. The method according to claim 2, characterized in that, The step of inputting a random self-learning sequence into the decoder of the prediction model, extracting features using the decoder to obtain a self-learning feature sequence, and using the historical trajectory feature sequence to guide the self-learning feature sequence to generate a predicted future trajectory sequence includes: A second transformed sequence is obtained by performing a nonlinear transformation on the random self-learning sequence using a feedforward neural network. The self-attention mechanism is used to calculate the correlation between each element of the second transformation sequence and all elements of the second transformation sequence to obtain a self-learning feature sequence; The self-learning feature sequence and the historical trajectory feature sequence are cross-attentioned and fused using a cross-attention mechanism to obtain a predicted future trajectory sequence.

5. The method according to any one of claims 1-4, characterized in that, The process of processing the reporting information within the current sliding window to obtain the current reporting sequence includes: Calculate the reporting velocity and acceleration of all reporting points within the current sliding window; The coordinates, velocity, and acceleration of all reporting points within the current sliding window are concatenated to form the current reporting sequence.

6. The method according to any one of claims 1-4, characterized in that, The process of processing the reporting information within the current sliding window to obtain the current reporting sequence includes: Calculate the reporting velocity, reporting acceleration, and tilt value for all reporting points within the current sliding window; The coordinates, velocity, acceleration, and tilt value of all reporting points within the current sliding window are concatenated to form the current reporting sequence.

7. The method according to any one of claims 1-4, characterized in that, Before inputting the current reporting sequence into the prediction model, the method further includes rounding each element in the current reporting sequence to form a current reporting sequence in which each element is an integer.

8. A writing trajectory prediction device for a touch display device, characterized in that, The device includes: The acquisition module is used to acquire multiple reporting point information of the touch object that is touching the touch display device, the reporting point information including the reporting point coordinates and the reporting time; The processing module is used to process the reporting information in the current sliding window to obtain the current reporting sequence; The learning module is used to input the current reporting sequence into the prediction model and predict at least one future reporting coordinate; the prediction model is trained on a deep learning-based transformer model using training data, and the prediction model is used to generate the at least one future reporting coordinate based on the current reporting sequence. The rendering module is used to render the trajectory of the last report point coordinates and at least one future report point coordinates.

9. A touch display device, characterized in that, The method includes a memory and at least one processor; wherein the memory is used to store one or more computer instructions, which are executed by the processor to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by a processor, the computer instructions implement the method as described in any one of claims 1-7.

Images