Handwritten text prediction method, related device and medium

Abstract

The invention provides a handwritten text prediction method, a related device and a medium. The handwritten text prediction method comprises the following steps: acquiring a target handwritten text track; sampling the target handwritten text track in a time sequence to obtain a time sequence sampling coordinate; extracting time sequence space features of the target handwritten text track from the time sequence sampling coordinates; obtaining previous text information of the target handwritten text track from an identification result of a handwritten text input before the target handwritten text track; and inputting the time sequence spatial features and the previous text information into a fusion prediction model to obtain predicted text information. According to the embodiment of the invention, the to-be-input text can be predicted with high accuracy according to the input handwritten text track. The embodiment of the invention can be applied to scenes such as handwriting input and text prediction.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to a handwritten text prediction method, related devices, and media. Background Art

[0002] At present, deep neural networks are widely used in handwritten text recognition. Generally, after the subject writes one or more complete words on the touch screen, the deep neural network recognizes the trajectory of the one or more complete words as a whole, and displays the recognition results on the screen. At present, there are two main technologies for handwritten text recognition. One technology is to extract the static features of the strokes from each handwritten stroke, and at the same time extract the stroke direction features from the handwritten text trajectory, and recognize the handwritten text through these two features. The other technology is to connect the sequence modeling architecture (such as RNN) in series on the basis of the former technology. Both technologies are recognition methods for input handwritten text, not prediction methods. The existing technology cannot predict the text to be input based on the trajectory of the handwritten text that has been input, and the recognition accuracy of the handwritten text of the two technologies is not high. Summary of the Invention

[0003] The embodiments of the present disclosure provide a handwritten text prediction method, related devices, and media, which can predict the text to be input based on the trajectory of the handwritten text that has been input with high accuracy.

[0004] According to one aspect of the present disclosure, a handwritten text prediction method is provided, comprising:

[0005] Get the target handwritten text trajectory;

[0006] Sampling the target handwritten text trajectory in time sequence to obtain time series sampling coordinates;

[0007] Extracting temporal spatial features of the target handwritten text trajectory from the temporal sampling coordinates;

[0008] Acquire context information of the target handwritten text track from a recognition result of a handwritten text input before the target handwritten text track;

[0009] The temporal spatial features and the above information are input into a fusion prediction model to obtain predicted text information.

[0010] According to one aspect of the present disclosure, a handwritten text prediction device is provided, comprising:

[0011] A first acquisition unit, configured to acquire a target handwritten text trajectory;

[0012] A first sampling unit is used to sample the target handwritten text trajectory in time sequence to obtain time series sampling coordinates;

[0013] A first extraction unit, configured to extract temporal spatial features of the target handwritten text trajectory from the temporal sampling coordinates;

[0014] A second acquiring unit, configured to acquire context information of the target handwritten text trajectory from a recognition result of a handwritten text input before the target handwritten text trajectory;

[0015] The first input unit is used to input the temporal spatial features and the above information into a fusion prediction model to obtain predicted text information.

[0016] Optionally, the first acquiring unit is specifically configured to:

[0017] Performing text recognition on the input handwritten text trajectory after the handwritten text input starts, and obtaining the input text recognition result;

[0018] Inputting the input text recognition result into a phrase segmentation model to obtain segmented phrases;

[0019] The input handwritten text trajectory starting from the starting time point of the phrase closest to the current time point is used as the target handwritten text trajectory.

[0020] Optionally, the first acquiring unit is further configured to:

[0021] Performing breakpoint recognition on the handwritten text track that has been input after the handwritten text input starts, and obtaining the recognized breakpoints;

[0022] Determine a first breakpoint from the identified breakpoints, where the first breakpoint is the breakpoint closest to the current time point among the breakpoints whose disconnection time exceeds the first duration;

[0023] The input handwritten text trajectory starting from the first breakpoint is used as the target handwritten text trajectory.

[0024] Optionally, the target handwritten text trajectory is formed by movement of a finger on the touch screen;

[0025] The first sampling unit is specifically configured to:

[0026] Acquire the moving speed of the finger on the touch screen;

[0027] determining a sampling frequency based on the movement speed;

[0028] Based on the sampling frequency, the target handwritten text trajectory is sampled to obtain the time series sampling coordinates.

[0029] Optionally, the time series sampling coordinates are coordinates of a plurality of sampling points in time sequence;

[0030] The first extraction unit is specifically configured to:

[0031] Based on the time series sampling coordinates, acquiring sampling point features of the target handwritten text trajectory;

[0032] The sampling point features are input into a temporal spatial feature extraction model to obtain the temporal spatial features.

[0033] Optionally, the handwritten text prediction device further includes:

[0034] A third acquiring unit is configured to acquire a set of sample handwritten text trajectories, wherein the sample handwritten text trajectories in the set of sample handwritten text trajectories have a sample previous context label and a sample next context label;

[0035] A second sampling unit is used to sample the sample handwritten text trajectory in time sequence to obtain sample time series sampling coordinates;

[0036] A second extraction unit, configured to extract sample temporal spatial features of the sample handwritten text trajectory from the sample temporal sampling coordinates;

[0037] A second input unit is configured to input the sample temporal spatial features and the sample context label into a fusion prediction model to obtain a predicted sample context;

[0038] A loss calculation unit is used to calculate a loss function based on the difference between the predicted sample context and the sample context label, and jointly train the temporal spatial feature extraction model and the fusion prediction model through the loss function.

[0039] Optionally, the handwritten text prediction device further includes:

[0040] a selection unit, configured to select adjacent stroke pairs to be exchanged from the sample handwritten text trajectories in the sample handwritten text trajectory set;

[0041] an interchanging unit, configured to interchange the adjacent strokes in the adjacent stroke pairs to be interchanged, to obtain an interchanged handwritten text trajectory;

[0042] The first expansion unit is configured to expand the exchanged handwritten text trajectory into the sample handwritten text trajectory set, wherein the exchanged handwritten text trajectory has the same sample previous-context label and the same sample next-context label as the sample handwritten text trajectory.

[0043] Optionally, the handwritten text prediction device further includes:

[0044] a rotation unit, configured to rotate the sample handwritten text trajectories in the sample handwritten text trajectory set by a plurality of rotation angles to obtain a plurality of rotated handwritten text trajectories, wherein the plurality of rotation angles are in an arithmetic progression;

[0045] The second expansion unit is configured to expand a plurality of rotated handwritten text trajectories into the sample handwritten text trajectory set, wherein the rotated handwritten text trajectories have the same sample upper context label and the sample lower context label as the sample handwritten text trajectories.

[0046] Optionally, the second sampling unit is specifically configured to: sample the sample handwritten text trajectory in time sequence according to multiple sampling frequencies to obtain multiple groups of sample time sequence sampling coordinates;

[0047] The second input unit is specifically configured to: for each set of sample time series sampling coordinates, input the sample time series spatial features and the sample context labels corresponding to the set of sample time series sampling coordinates into the fusion prediction model to obtain the sample context predicted for the set of sample time series sampling coordinates;

[0048] The loss calculation unit is specifically configured to calculate a loss function corresponding to the set of sample time series sampling coordinates based on a difference between the sample context predicted for the set of sample time series sampling coordinates and the sample context label.

[0049] Optionally, the sample time series sampling coordinates are coordinates of a plurality of sample sampling points in time sequence;

[0050] The second extraction unit is specifically configured to:

[0051] Based on the sample time series sampling coordinates, obtaining sample sampling point features of the sample handwritten text trajectory;

[0052] The sample sampling point features are input into the temporal spatial feature extraction model to obtain the sample temporal spatial features.

[0053] Optionally, the second extraction unit is further configured to:

[0054] determining a sample normalized displacement difference of the sample sampling point based on a third coordinate of the sample sampling point and a fourth coordinate of a sample sampling point preceding the sample sampling point;

[0055] Obtaining the sample lifting and dropping pen identifier of the sample sampling point;

[0056] Obtaining a first sampling point identifier of a sample stroke, where the first sampling point identifier indicates whether the sample sampling point is the first sample sampling point in the stroke where the sample sampling point is located;

[0057] The sample sampling point features of the sample handwritten text trajectory are generated based on the sample normalized displacement difference, the sample pen lift and drop identifiers, and the sample stroke first sampling point identifiers.

[0058] Optionally, the third coordinate includes a third abscissa and a third ordinate, and the fourth coordinate includes a fourth abscissa and a fourth ordinate;

[0059] The second extraction unit is further configured to:

[0060] determining a third difference between the third abscissa of the sample sampling point and the fourth abscissa of a sample sampling point preceding the sample sampling point, and a fourth difference between the third ordinate of the sample sampling point and the fourth ordinate of the sample sampling point preceding the sample sampling point;

[0061] Obtaining a maximum value of a sample vertical coordinate and a minimum value of a sample vertical coordinate of the sample handwritten text trajectory in the sample handwritten text trajectory set;

[0062] Determine the sample ordinate span based on the sample ordinate maximum value and the sample ordinate minimum value;

[0063] Determine a sample normalized horizontal coordinate displacement differential of the sample sampling point based on the third difference and the sample vertical coordinate span, and determine a sample normalized vertical coordinate displacement differential of the sampling point based on the fourth difference and the sample vertical coordinate span;

[0064] The sample normalized displacement difference of the sample sampling point is determined based on the sample normalized horizontal coordinate displacement difference and the sample normalized vertical coordinate displacement difference.

[0065] Optionally, the first extraction unit is further configured to:

[0066] Determining a normalized displacement difference of the sampling point based on a first coordinate of the sampling point and a second coordinate of a sampling point preceding the sampling point;

[0067] Obtaining the pen lift and drop identifier of the sampling point;

[0068] Obtaining a stroke first sampling point identifier, where the stroke first sampling point identifier indicates whether the sampling point is the first sampling point in the stroke where the sampling point is located;

[0069] The sampling point features of the target handwritten text trajectory are generated based on the normalized displacement difference, the pen lift and pen drop identifiers, and the stroke first sampling point identifier.

[0070] Optionally, the first coordinate includes a first abscissa and a second ordinate, and the second coordinate includes a second abscissa and a second ordinate;

[0071] The first extraction unit is further configured to:

[0072] Determine a first difference between the first horizontal coordinate of the sampling point and the second horizontal coordinate of a sampling point preceding the sampling point, and a second difference between the first vertical coordinate of the sampling point and the second vertical coordinate of the sampling point preceding the sampling point;

[0073] Obtaining a maximum value and a minimum value of the ordinate of the target handwritten text trajectory;

[0074] Determining a vertical coordinate span based on the vertical coordinate maximum value and the vertical coordinate minimum value;

[0075] Determining a normalized horizontal displacement differential of the sampling point based on the first difference and the vertical coordinate span, and determining a normalized vertical displacement differential of the sampling point based on the second difference and the vertical coordinate span;

[0076] Based on the normalized horizontal coordinate displacement difference and the normalized vertical coordinate displacement difference, the normalized displacement difference of the sampling point is determined.

[0077] Optionally, the first extraction unit is further configured to:

[0078] Acquire a first pressure at the sampling point and a second pressure at a sampling point preceding the sampling point;

[0079] If the first pressure is less than the second pressure, determining the pen-lifting / lowering flag as a first value indicating pen-lifting;

[0080] If the first pressure is not less than the second pressure, the pen-up / pen-down flag is determined as a second value indicating pen-down.

[0081] Optionally, the handwritten text prediction device further includes:

[0082] an interruption setting unit, configured to set an interruption sampling point at a position where a stroke of the target handwritten text trajectory is interrupted, wherein the abscissa of the interruption sampling point is outside a value range of the abscissa of the target handwritten text trajectory, and the ordinate of the interruption sampling point is outside a value range of the ordinate of the target handwritten text trajectory;

[0083] an adding unit, configured to add the interrupted sampling point to the plurality of sampling points;

[0084] The first extraction unit is further configured to: set the stroke first sampling point identifier to a third value if the sampling point preceding the sampling point is the interruption sampling point; and set the stroke first sampling point identifier to a fourth value if the sampling point preceding the sampling point is not the interruption sampling point.

[0085] Optionally, the temporal spatial feature extraction model includes a long short-term memory network and a convolutional layer;

[0086] The first extraction unit is further configured to:

[0087] Inputting the sampling point features into the long short-term memory network to obtain a first intermediate vector;

[0088] The first intermediate vector is input into the convolutional layer to obtain the temporal spatial feature.

[0089] Optionally, the temporal spatial feature extraction model further includes a first linear layer and a maximum pooling layer;

[0090] The first extraction unit is further configured to:

[0091] Inputting the first intermediate vector into the first linear layer to obtain a second intermediate vector;

[0092] Inputting the second intermediate vector into the convolutional layer to obtain a third intermediate vector;

[0093] The third intermediate vector is input into the maximum pooling layer to obtain temporal spatial features.

[0094] Optionally, the first input unit is specifically configured to:

[0095] Determine an embedding vector corresponding to the above information;

[0096] Adding the position encoding information of each character of the above information to the embedding vector;

[0097] The embedding vector and the temporal spatial feature are input into the fusion prediction model to obtain the predicted text information.

[0098] Optionally, the fusion prediction model includes a multi-head attention model and a fusion layer;

[0099] The first input unit is further configured to:

[0100] Inputting the embedding vector into the multi-head attention model to obtain an attention vector;

[0101] Inputting the attention vector and the temporal spatial feature into a fusion layer to obtain a fusion vector;

[0102] The fusion vector is input into the decision layer to obtain the predicted text information.

[0103] Optionally, the first input unit is further configured to:

[0104] Superimposing the attention vector and the embedding vector to obtain a first superimposed vector;

[0105] Normalizing the first superposition vector to obtain a first normalized vector;

[0106] The first normalized vector and the temporal spatial feature are input into the fusion layer to obtain the fusion vector.

[0107] Optionally, the first input unit is further configured to:

[0108] Inputting the first normalized vector and the temporal spatial feature into the fusion layer to obtain a fourth intermediate vector;

[0109] Superimposing the fourth intermediate vector, the first normalized vector, and the temporal spatial feature to obtain a second superimposed vector;

[0110] Normalizing the second superposition vector to obtain a second normalized vector;

[0111] Inputting the second normalized vector into the second linear layer to obtain a fifth intermediate vector;

[0112] Superimposing the second normalized vector and the fifth intermediate vector to obtain a third superimposed vector;

[0113] The third superposition vector is normalized to obtain the fusion vector.

[0114] Optionally, the first input unit is further configured to:

[0115] Inputting the fused vector into a third linear layer to obtain a sixth intermediate vector;

[0116] Inputting the sixth intermediate vector into the decision layer to obtain probabilities of multiple candidate text information;

[0117] The predicted text information is determined based on the probabilities of the plurality of candidate text information.

[0118] According to one aspect of the present disclosure, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the handwritten text prediction method as described above when executing the computer program.

[0119] According to one aspect of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the handwritten text prediction method as described above is implemented.

[0120] According to one aspect of the present disclosure, a computer program product is provided. The computer program product includes a computer program. The computer program is read and executed by a processor of a computer device, so that the computer device executes the handwritten text prediction method as described above.

[0121] The embodiment of the present disclosure samples the target handwritten text trajectory in time sequence to obtain time-series sampling coordinates, and extracts the time-series spatial features of the target handwritten text trajectory from the time-series sampling coordinates. This feature is different from the static stroke feature in the prior art, which reflects the order in which the strokes are input. It is also different from the stroke direction feature in the prior art, because it not only reflects the spatial extension relationship of the strokes, but also reflects the temporal order of the strokes. In addition, the embodiment of the present disclosure does not input the spatial features and context information into different models separately as in the prior art, but uniformly inputs the time-series spatial features and the above information into the fusion prediction model to obtain predicted text information, thereby reducing cascade errors and improving prediction accuracy. Unlike the prior art that can only recognize the text information that has been input, since it fully learns the order features of the strokes in the text trajectory and the context features of multiple characters, the embodiment of the present disclosure can accurately predict the text to be input based on the input handwritten text trajectory through the relationship between the strokes and the context relationship.

[0122] Other features and advantages of the present disclosure will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present disclosure. The purposes and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0123] The accompanying drawings are used to provide a further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation to the technical solution of the present disclosure.

[0124] Figure 1 is a system architecture diagram of the handwritten text prediction method according to an embodiment of the present disclosure;

[0125] Figures 2A to 2C This is a schematic diagram of an interface for generating predicted text information based on a target handwritten text trajectory in a handwritten text prediction application provided by an embodiment of the present disclosure.

[0126] Figure 3 is a main flow chart of a handwritten text prediction method according to an embodiment of the present disclosure;

[0127] Figure 4 is a general implementation structure diagram of a handwritten text prediction method according to an embodiment of the present disclosure;

[0128] Figure 5A and Figure 5B These are two implementation structure diagrams of text recognition methods;

[0129] Figure 6 yes Figure 3 A flow chart of obtaining the target handwritten text trajectory in step 310;

[0130] Figure 7 yes Figure 6 Obtain a schematic diagram of the target handwritten text trajectory;

[0131] Figure 8 yes Figure 3 Another flow chart of obtaining the target handwritten text trajectory in step 310;

[0132] Figure 9 yes Figure 8 Obtain a schematic diagram of the target handwritten text trajectory;

[0133] Figure 10 yes Figure 3 A flow chart of obtaining the time series sampling coordinates in step 320;

[0134] Figure 11 yes Figure 3 A flow chart of extracting temporal spatial features in step 330;

[0135] Figure 12 yes Figure 11 A flow chart of obtaining sampling point features in step 1110;

[0136] Figure 13 yes Figure 12 Get a schematic diagram of the sampling point features;

[0137] Figure 14 yes Figure 12 A flow chart of determining the normalized displacement difference in step 1210;

[0138] Figure 15 yes Figure 14 A schematic diagram for determining the normalized displacement difference;

[0139] Figure 16 yes Figure 12 A flow chart of obtaining the pen lift and drop identifier in step 1220;

[0140] Figure 17 yes Figure 16 Get a schematic diagram of the pen lift and drop mark;

[0141] Figure 18 yes Figure 12 A flow chart of obtaining the identification of the first sampling point of the stroke in step 1230;

[0142] Figure 19 yes Figure 11 A flow chart of obtaining temporal spatial features in step 1120;

[0143] Figure 20 yes Figure 19 Step 1920 is a flow chart for obtaining temporal spatial features;

[0144] Figure 21 yes Figure 3 A flow chart of obtaining predicted text information in step 350;

[0145] Figure 22 yes Figure 21 A flowchart of obtaining predicted text information in step 2130;

[0146] Figure 23 yes Figure 22 A flow chart of obtaining a fusion vector in step 2220;

[0147] Figure 24 yes Figure 23 A flow chart of obtaining a fusion vector in step 2330;

[0148] Figure 25 yes Figure 22 A flowchart of determining predicted text information in step 2230;

[0149] Figure 26 This is a main flow chart of the joint training method of the temporal spatial feature extraction model and the integration prediction model provided by the embodiments of the present disclosure;

[0150] Figure 27 This is a general implementation structure diagram of a temporal spatial feature extraction model and a joint training method for integrating the model into a prediction model according to an embodiment of the present disclosure;

[0151] Figure 28 is Figure 26 After step 2610, a flow chart for swapping adjacent strokes to be swapped is added;

[0152] Figure 29 yes Figure 28 A schematic diagram of swapping adjacent strokes to be swapped;

[0153] Figure 30 is Figure 26 After step 2610, a flowchart for rotating the sample handwritten text trajectory is added;

[0154] Figure 31 is Figure 30 A schematic diagram of rotating a sample handwritten text trajectory;

[0155] Figure 32 is another flow chart of a joint training method for a temporal spatial feature extraction model and an integrated prediction model provided by an embodiment of the present disclosure;

[0156] Figure 33 yes Figure 32 An implementation diagram of the joint training method for temporal spatial feature extraction model and integration into the prediction model;

[0157] Figure 34 yes Figure 26 A flow chart of extracting temporal spatial features of samples in step 2630;

[0158] Figure 35 yes Figure 34 A flow chart of obtaining the characteristics of the sample sampling points in step 3410;

[0159] Figure 36 yes Figure 35 A schematic diagram of obtaining sample sampling point characteristics;

[0160] Figure 37 yes Figure 35 A flow chart for determining the normalized displacement difference of samples;

[0161] Figure 38 yes Figure 37 A schematic diagram of determining the normalized displacement difference of samples;

[0162] Figure 39 Detailed implementation diagram of the handwritten text prediction method according to an embodiment of the present disclosure.

[0163] Figure 40 is a module diagram of a handwritten text prediction device according to an embodiment of the present disclosure;

[0164] Figure 41 According to the embodiment of the present disclosure Figure 3 The terminal structure diagram of the handwritten text prediction method shown;

[0165] Figure 42 According to the embodiment of the present disclosure Figure 3 The server structure diagram of the handwritten text prediction method shown in FIG. DETAILED DESCRIPTION

[0166] In order to make the purpose, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not intended to limit the present disclosure.

[0167] Before further explaining the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are explained. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations:

[0168] Deep Neural Network (DNN): A type of neural network in the field of Machine Learning (ML). It is a neural network with multiple hidden layers between the input layer and the output layer, which can represent complex functions with fewer parameters.

[0169] A recurrent neural network (RNN) is a type of recursive neural network that takes sequence data as input and recursively propagates along the evolution of the sequence, with all nodes connected in a chain-like fashion. RNNs are a sequence modeling architecture that meets the design criteria for sequence modeling, including processing variable-length sequences, tracking long-term dependencies, maintaining information about order, and sharing parameters across sequences.

[0170] Long Short-Term Memory (LSTM): A recurrent neural network capable of memorizing both long-term and short-term information. LSTMs incorporate a gate mechanism to control the flow and loss of features, enabling them to extract features from previous moments at the current moment, thereby resolving the long-term dependencies inherent in recurrent neural networks.

[0171] Handwritten text recognition uses a computer dictionary to quickly identify handwritten text on media such as paper or mobile phone screens. Handwritten text recognition essentially involves character classification, typically achieved through feature and feature matching.

[0172] At present, deep neural networks are widely used in handwritten text recognition. Generally, after the subject writes one or more complete words on the touch screen, the deep neural network recognizes the trajectory of the one or more complete words as a whole, and displays the recognition results on the screen. At present, there are two main technologies for handwritten text recognition. One technology is to extract the static features of the strokes from each handwritten stroke, and at the same time extract the stroke direction features from the handwritten text trajectory, and recognize the handwritten text through these two features. The other technology is to connect the sequence modeling architecture (such as RNN) in series based on the former technology. Both technologies are recognition methods for input handwritten text, not prediction methods. The relevant technologies cannot predict the text to be input based on the trajectory of the handwritten text that has been input, and the recognition accuracy of the handwritten text of the two technologies is not high.

[0173] System architecture and scenario description of the application of the embodiments of the present disclosure

[0174] Figure 1 1 is a diagram of a system architecture used in the handwritten text prediction method according to an embodiment of the present disclosure, which includes a target terminal 110, the Internet 120, a gateway 130, and a server 140.

[0175] The subject terminal 110 is a device used to display predicted text for viewing by the subject, and the subject can also input handwritten text through the subject terminal 110. It can take various forms, including desktop computers, laptops, PDAs (personal digital assistants), mobile phones, in-vehicle terminals, home theater terminals, dedicated terminals, and digital tablets. Furthermore, it can be a single device or a collection of multiple devices. For example, multiple devices connected via a local area network and sharing a common display device can collectively constitute a terminal. The terminal can also communicate with the Internet 120 via wired or wireless means to exchange data.

[0176] Server 140 is a computer system that provides handwritten text prediction services to terminal 110. Compared to terminal 110, server 140 has higher requirements for stability, security, and performance. Server 140 can be a single high-performance computer within a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines). Server 140 can also communicate with the internet 120 via wired or wireless means to exchange data.

[0177] The embodiments of the present disclosure can be applied in various scenarios, such as Figures 2A to 2C The figure shows an interface diagram for generating predicted text information based on a target handwritten text trajectory in a handwritten text prediction application.

[0178] Reference Figures 2A to 2C The subject can write in the handwriting area of ​​the handwriting prediction application, thereby inputting a handwritten text trajectory. The handwritten text trajectory input by the subject in the handwriting area is recognized and predicted, and then displayed in other areas of the handwriting prediction application.

[0179] Reference Figure 2A Based on the handwritten text trajectory input by the subject in the handwritten text prediction application, the handwritten text recognition result "Today's weather" is obtained. After that, the subject continues to input in the handwriting area.

[0180] The handwritten text recognition method in the related art cannot make predictions based on the input handwritten text trajectory. Figure 2A The handwriting area of ​​the handwriting prediction application shows the handwriting text trajectory. The handwriting text recognition method in the related art cannot predict the text to be input. Figure 2B It is only when the complete trajectory shown is obtained that "sunny" can be recognized.

[0181] However, based on Figure 2A the handwritten text trajectory shown in the handwritten area of the handwritten text prediction application in Figure 2C as shown, according to the handwritten text trajectory that has been input within the handwritten area, it is predicted that the character being input by the object is "sunny", and based on the previous text "Today's weather" and the character "sunny" being input, it is predicted that the next character to be input by the object is "bright".

[0182] By Figure 2B comparison with Figure 2C it can be clearly concluded that the handwritten text prediction method provided by the embodiments of the present disclosure can accurately predict the text to be input according to the handwritten text trajectory that has been input.

[0183] It should be understood that the above content only shows an illustration of some application scenarios of the present disclosure. The business scenarios to which the present disclosure can be applied may include, but are not limited to, the specific embodiments cited above.

[0184] General description of the embodiments of the present disclosure

[0185] It should be emphasized that the embodiments of the present disclosure can be applied to a variety of application scenarios, such as scenarios of handwritten recording of meeting summaries, handwritten recording of learning notes, etc. The handwritten text recognition methods in the related technologies are all methods for recognizing the handwritten text that has been input, rather than prediction methods. They cannot predict the text to be input according to the handwritten text trajectory that has been input, and the recognition accuracy is not high. Some embodiments of the present disclosure provide a handwritten text prediction method, related device and medium, which can accurately predict the text to be input according to the handwritten text trajectory that has been input.

[0186] The handwritten text prediction method is a method for processing the input target handwritten text trajectory to obtain the predicted text information. This handwritten text prediction method can be applied to various target handwritten text trajectories. The target handwritten text trajectory can be a single stroke, multiple strokes, the strokes corresponding to a single character, the strokes corresponding to a word, etc. The target handwritten text trajectory can also be the notes corresponding to a character with incorrect stroke order, the handwriting corresponding to an inclined character. In addition, the predicted text information can be the character that the object is currently inputting, as well as the word, phrase, sentence, etc. that the object will input. This handwritten text prediction method can accurately predict the text to be input according to the handwritten text trajectory that has been input, through the relationship between strokes and the context relationship.

[0187] The handwritten text prediction method according to an embodiment of the present disclosure can be executed on the object terminal 130, or on the server 140, or part of it can be executed on the object terminal 130 and the other part on the server 140.

[0188] As Figure 3 shown, according to an embodiment of the present disclosure, the handwritten text prediction method includes:

[0189] Step 310, obtaining a target handwritten text trajectory;

[0190] Step 320, sampling the target handwritten text trajectory in chronological order to obtain chronological sampling coordinates;

[0191] Step 330, extracting the chronological spatial features of the target handwritten text trajectory from the chronological sampling coordinates;

[0192] Step 340, obtaining the context information of the target handwritten text trajectory from the recognition result of the handwritten text input before the target handwritten text trajectory;

[0193] Step 350, inputting the chronological spatial features and the context information into a fusion prediction model to obtain the predicted text information.

[0194] The following briefly describes Steps 310 to 350.

[0195] In Step 310, a target handwritten text trajectory is obtained. The target handwritten text trajectory refers to the part of the handwritten text trajectory that is closest to the current time point among the input handwritten text trajectories after the start of handwritten text input, and it can also be all the input handwritten text trajectories after the start of handwritten text input.

[0196] The target handwritten text trajectory can be the strokes corresponding to the character being input by the object at the current time point, and the number of strokes is at least 1. For example Figure 2A the "day" shown in the handwritten area. The target handwritten text trajectory can also be the strokes corresponding to the word, sentence or paragraph being input by the object at the current time point. For example, when the target handwritten text trajectory is the strokes of the word being input by the object at the current time point, Figure 2A the target handwritten text trajectory corresponding to the interface shown is "day"; when the target handwritten text trajectory is the strokes of the sentence being input by the object at the current time point, Figure 2A the target handwritten text trajectory corresponding to the interface shown is "Today's weather day"; when the target handwritten text trajectory is the strokes of the paragraph being input by the object at the current time point, Figure 2A the target handwritten text trajectory corresponding to the interface shown is "Today's weather day". In addition, when the target handwritten text trajectory is all the input handwritten text trajectories after the start of handwritten text input, Figure 2AThe target handwritten text trajectory corresponding to the interface shown is “XX…XX weather today”.

[0197] It should be noted that the target text track can be Chinese character strokes, Arabic numerals, English, etc. Specifically, the language corresponding to the target text track can be set as needed.

[0198] In addition, the target handwritten text trajectory can be obtained in real time, at fixed intervals, or when the pen pressure on a terminal, such as a mobile phone touch screen, a digital tablet, or a handwriting tablet, remains at zero for a preset time. The specific method for obtaining the target handwritten text trajectory can be set as needed.

[0199] In step 320, the target handwritten text trajectory is sampled in time sequence to obtain time-series sampling coordinates. Time-series sampling coordinates refer to the coordinates of multiple points in the target text trajectory that are consecutive in time. Time-series sampling coordinates contain not only the position information of each point, but also the time sequence information of each point. The time sequence information can reflect the order of multiple strokes corresponding to the target handwritten text trajectory and the direction of each stroke.

[0200] The target handwritten text trajectory is sampled in time sequence. This means that new handwritten text trajectory points on the terminal are acquired at regular intervals. This represents the target handwritten text trajectory using the coordinates of multiple chronologically arranged handwritten text trajectory points, known as the time-series sampling coordinates. In other words, the acquisition and sampling of the target handwritten text trajectory are performed simultaneously.

[0201] In addition, the time series sampling coordinates can be expressed as f = (f1, f2, ..., ft), where ft = (xt, yt) is the coordinate point of the target handwritten text trajectory at the tth time.

[0202] Different terminal devices have different screen sizes and pixel counts. Therefore, the coordinate axes corresponding to the time-series sampling coordinates need to be determined based on the terminal device, thereby determining the time-series sampling coordinates corresponding to each sampling point in the target handwritten text trajectory. In addition, the coordinate axes corresponding to different terminal devices can be converted to the same standard, thereby converting the handwritten text trajectories input from different terminal devices to the same standard coordinate axes for processing.

[0203] In step 330, the temporal spatial features of the target handwritten text trajectory are extracted from the temporal sampling coordinates. Temporal spatial features refer to the features of the temporal sampling coordinates, which can represent the spatial and temporal features of the target handwritten text trajectory. Specifically, when the temporal sampling coordinates are f = (f1, f2, …, ft), the temporal spatial features can be represented in the form of a matrix or a vector. When the temporal spatial features are represented in the form of a matrix, each row of the matrix corresponds one-to-one to each coordinate point, and the elements of each row are used to represent the position of one of the temporal sampling coordinates and the sampling time corresponding to the temporal sampling coordinate. For example, the first row of the matrix is ​​set with three elements, and the three elements are used to represent the horizontal coordinate, vertical coordinate, and first time of the coordinate point at the first time, respectively. When the temporal spatial features are expressed in the form of a vector, the horizontal coordinates, vertical coordinates, and sampling times of multiple coordinate points can be converted into vector form and spliced ​​together. The resulting temporal spatial features are [! H1,! R1,! T1, …! Ht,! Rt,! Tt], where "!H1" represents the horizontal coordinate of the coordinate point at the first moment, "!R1" represents the vertical coordinate of the coordinate point at the first moment, "!T1" represents the first moment, "!Ht" represents the horizontal coordinate of the coordinate point at the tth moment, "!Rt" represents the vertical coordinate of the coordinate point at the tth moment, and "!Tt" represents the tth moment.

[0204] In addition, in addition to the spatial features and temporal features of the target handwritten text trajectory, the temporal spatial features can also include other content, such as the start and end marks of each stroke, so as to divide the strokes corresponding to the target handwritten text trajectory, thereby determining the multiple strokes corresponding to the target handwritten text trajectory, and then achieving more accurate prediction of the text that the object is expected to input.

[0205] The extraction of temporal spatial features can be performed in a variety of ways. For example, the temporal sampling coordinates are used as the input of the encoder, and the output of the encoder is the temporal spatial features of the target handwritten text trajectory. Alternatively, the temporal sampling coordinates can be extracted through a neural network to obtain the temporal spatial features of the target handwritten text trajectory.

[0206] In step 340, context information of the target handwritten text track is obtained from the recognition results of the handwritten text input before the target handwritten text track. The context information of the target handwritten text track refers to the text information that has been recognized and determined in the handwritten text input before the target handwritten text track, and this text information is adjacent to the target handwritten text track.

[0207] The above context information is determined based on the target handwritten text trajectory, which can be a word, a phrase, a sentence, a paragraph, or even all the text content before the target handwritten text trajectory. In order to improve the accuracy of handwritten text prediction, the more above context information corresponding to the target handwritten text trajectory, the better. In order to improve the processing speed of handwritten text prediction, the above context information corresponding to the target handwritten text trajectory needs to be as little as possible. Therefore, in the actual processing process, the above context information corresponding to the target handwritten text trajectory can be determined based on the semantic relevance between the text contents. For example, Figure 2A The recognition result of the handwritten text input before the target handwritten text track in is "today's weather". After semantic correlation processing of "today's weather", it can be determined that "today" and "weather" are two words respectively, and the semantic correlation between the two words is small. It can be determined that the word corresponding to the target handwritten text track is a modifier of "weather". Therefore, Figure 2A The target text track shown above corresponds to the information "weather". In addition, Figure 2A The context information corresponding to the target text track shown may also be “today’s weather”, “air”, etc.

[0208] It should be noted that, when the target handwritten text trajectory is a single stroke, the context information corresponding to the target handwritten text trajectory also includes other input strokes of the character corresponding to the stroke.

[0209] In step 350, the temporal spatial features and the above context information are input into the fusion prediction model to obtain predicted text information. The predicted text information refers to the text information currently being input by the subject corresponding to the target handwritten text trajectory. The text information can be a character, word, sentence, etc.

[0210] A fusion prediction model is one that processes the temporal spatial features and contextual information corresponding to the target handwritten text trajectory, thereby predicting the text to be entered based on the input handwritten text trajectory, the relationships between strokes, and the contextual relationships. The fusion prediction model can be a deep neural network model or another model, such as an encoder. The disclosed embodiment constructs a fusion prediction model based on a deep neural network model, which can further improve the fusion prediction model's prediction accuracy for text information compared to other models.

[0211] Reference Figure 4, the embodiment of the present disclosure first obtains the target handwritten text trajectory, and samples the target handwritten text trajectory in time sequence to obtain time series sampling coordinates. The time series sampling coordinates contain the position information and time series information of each sampling point corresponding to the target handwritten text trajectory. Feature extraction is performed on the time series sampling coordinates to obtain the time series space features corresponding to the target handwritten text trajectory. In addition, based on the recognition result of the handwritten text input before the target handwritten text trajectory, the above information corresponding to the target handwritten text trajectory can be determined, and the above information and the time series space features are used as the input of the fusion prediction model to obtain the predicted text information.

[0212] Figure 5A and Figure 5B This is a structural diagram corresponding to the handwritten text recognition method in the related art. Figure 5A As shown in the figure, this text recognition method extracts static stroke features from each handwritten stroke and extracts stroke direction features from the handwritten text trajectory. It recognizes handwritten text by these two features. It is only suitable for single word recognition and is prone to error propagation when processing continuous handwritten characters. It is certain that Figure 5A Compared with the static features of the strokes shown, the temporal space features of the embodiment of the present disclosure can reflect the order of input of the strokes, and compared with the stroke direction features, the temporal space features can not only reflect the spatial extension relationship of the strokes, but also reflect the temporal order of the strokes. Figure 5B The handwritten text recognition method shown is in Figure 5A Based on the cascade sequence modeling architecture, the text trajectory is processed in sequence through two models. However, the spatial features corresponding to the handwritten text trajectory, namely the static features of the strokes and the direction features of the strokes, and the processing of the above information are located in two independent modules, which is prone to cascading errors and cannot well integrate the spatial features and the above information, resulting in poor recognition accuracy. Figure 5B Compared with the model shown in FIG, the embodiment of the present disclosure integrates the temporal spatial features and the above information into the fusion prediction model, which can reduce cascade errors and improve prediction accuracy. Figure 5A and Figure 5B The handwritten text recognition methods shown can only recognize the input handwritten text, that is, the complete text trajectory corresponding to the character or word must be input before recognition can be performed. However, the present application can predict the text to be input based on the input target handwritten text trajectory.

[0213] The embodiment of steps 310 to 350 above samples the target handwritten text trajectory in time sequence to obtain time-series sampling coordinates, and extracts the time-series spatial features of the target handwritten text trajectory from the time-series sampling coordinates. This feature is different from the static stroke features in the prior art, which reflects the order in which the strokes are input. It is also different from the stroke direction features in the prior art because it not only reflects the spatial extension relationship of the strokes, but also reflects the temporal order of the strokes. In addition, the embodiment of the present disclosure does not input the spatial features and context information into different models separately as in the prior art, but uniformly inputs the time-series spatial features and the above information into the fusion prediction model to obtain the predicted text information, thereby reducing cascade errors and improving prediction accuracy. Unlike the prior art that can only recognize the text information that has been input, because it fully learns the order features of the strokes in the text trajectory and the context features of multiple characters, the embodiment of the present disclosure can accurately predict the text to be input based on the input handwritten text trajectory through the relationship between the strokes and the context relationship.

[0214] The above is a general description of steps 310 to 350 . The specific implementation process of steps 310 to 350 will be described in detail below.

[0215] Detailed description of step 310

[0216] In step 310 , the target handwritten text trajectory is obtained.

[0217] In one embodiment, referring to Figure 6 , step 310 includes:

[0218] Step 610: Perform text recognition on the handwritten text track after the handwritten text input starts, and obtain the input text recognition result;

[0219] Step 620: Input the input text recognition result into the phrase segmentation model to obtain segmented phrases;

[0220] Step 630: Use the input handwritten text trajectory starting from the starting time point of the most recent meaning group before the current time point as the target handwritten text trajectory.

[0221] Steps 610 to 630 are described in detail below.

[0222] In step 610, text recognition is performed on the input handwritten text trajectory after the start of handwritten text input to obtain the recognized result of the input text. The input handwritten text trajectory refers to the text trajectory that the object has input at the terminal, and can also be understood as all the handwritten text trajectories input by the object before the current time point. The recognized result of the input text refers to the text information determined by performing text recognition on the handwritten text trajectory, and can also be the text information predicted for the handwritten text trajectory at a previous moment and determined by the object. The text information in the recognized result of the input text is correct after being determined by the object.

[0223] For the case where the object continuously writes multiple text contents, the start of handwritten text input refers to the start of handwritten input for the current handwritten article, handwritten meeting minutes, etc.

[0224] It should be noted that when the semantic connection between the upper and lower sentences and paragraphs in a text content is relatively small, several sentences or paragraphs that are the closest to the current time point and have a semantic relevance greater than a preset threshold between the context contents can be used as the recognized result of the input text. This setting of the recognized result of the input text can reduce the processing volume of the sense group segmentation model, thereby reducing the processing overhead and improving the prediction speed of handwritten text.

[0225] In step 620, the recognized result of the input text is input into the sense group segmentation model to obtain the segmented sense groups. A sense group is a concept in linguistics, which refers to a series of relatively independent semantic units obtained by dividing text information according to the needs of meaning and structure. These semantic units are usually complete in semantics and are grammatically related and cannot be further subdivided. Each sense group consists of a group of closely related words or phrases, and the relationship between them can be logical coordination, progression, or other semantic relationships. The sense group segmentation model refers to a model that can divide text information into multiple sense groups.

[0226] The embodiments of the present disclosure can perform word segmentation, sentence segmentation, or paragraph segmentation on the recognized result of the input text through the sense group segmentation model, and the multiple sense groups obtained correspond to the sense group segmentation method. For example, referring to Figure 7 , the sense group segmentation method corresponding to this sense group segmentation model is word segmentation. The recognized result of the input text is "The weather is sunny today" and some strokes of the character "lang". The recognized result of the input text is input into the sense group segmentation model to obtain multiple segmented sense groups, namely sense group 1 "today", sense group 2 "weather", sense group 3 "sunny", and some strokes of the character "lang".

[0227] In addition, when the phrase segmentation mode is sentence segmentation, the phrase segmentation model can search for punctuation marks such as periods and semicolons in the input text recognition results, thereby segmenting the input text recognition results into multiple sentence phrases. When the phrase segmentation mode is paragraph segmentation, the phrase segmentation model can search for line breaks in the input text recognition results, thereby segmenting the input text recognition results into multiple paragraph phrases.

[0228] In step 630, the input handwritten text trajectory starting from the start time of the closest phrase before the current time point is used as the target handwritten text trajectory. The target handwritten text trajectory is the character, word, sentence, or paragraph currently being input by the subject. Therefore, the input handwritten text trajectory corresponding to the closest phrase to the current time point is used as the target handwritten text trajectory.

[0229] The starting time point of a phrase refers to the time point when the first stroke corresponding to the phrase is started, and the input handwritten text trajectory starting from the starting time point of the phrase refers to the handwritten text trajectory corresponding to the phrase.

[0230] Reference Figure 3 , the input text recognition results are segmented by the phrase segmentation model to obtain three phrases. The three phrases are arranged in chronological order as phrase 1, phrase 2, and phrase 3. Then phrase 3 is the closest phrase between the current time points, and the input handwritten text trajectory corresponding to phrase 3 is the target handwritten text trajectory.

[0231] The above steps 610 to 630 first perform text recognition on the input handwritten text trajectory after the start of handwritten text input to obtain the input text recognition result, and then process the input text recognition result through the phrase segmentation model to obtain multiple segmented phrases. The phrases are determined based on semantics, so that the multiple characters corresponding to the words, sentences, and paragraphs corresponding to the obtained target handwritten text trajectory are semantically related, so that the handwritten text trajectory to be input can be determined based on the input handwritten text trajectory, thereby improving the accuracy of handwritten text prediction.

[0232] In another embodiment, referring to Figure 8 , step 310 includes:

[0233] Step 810: performing breakpoint recognition on the handwritten text track after the handwritten text input starts, and obtaining recognized breakpoints;

[0234] Step 820: Determine a first breakpoint from the identified breakpoints, where the first breakpoint is the breakpoint closest to the current time point among the breakpoints whose disconnection time exceeds the first duration.

[0235] Step 830: Use the input handwritten text trajectory starting from the first breakpoint as the target handwritten text trajectory.

[0236] The following provides a detailed description of steps 810 to 830.

[0237] In step 810, breakpoint recognition is performed on the input handwritten text trajectory after the start of the handwritten text input, and the recognized breakpoints are obtained. A breakpoint refers to the interruption position between two adjacent strokes in the input handwritten text trajectory. Between two breakpoints adjacent in chronological order is a completed stroke or a connected writing trajectory of multiple strokes.

[0238] It should be noted that handwritten text prediction continues as the object writes. Correspondingly, breakpoint recognition of the input handwritten text trajectory also continues. Thus, the strokes between the multiple breakpoints obtained by breakpoint recognition during the previous handwritten text prediction are determined at the current time point. Therefore, to reduce the data processing volume of breakpoint recognition, embodiments of the present disclosure can obtain the input handwritten text trajectory starting from the breakpoint closest to the current time point obtained last time for breakpoint recognition. Additionally, the input handwritten text trajectory corresponding to breakpoint recognition can also be obtained at fixed time intervals, and each time the obtained input handwritten text trajectory is the new handwritten text trajectory generated within the corresponding time interval.

[0239] Refer to Figure 9 , the input text trajectory is "jin er", where some strokes in "jin" and "er" are in a connected writing trajectory. Breakpoint recognition is performed on the input text trajectory to obtain breakpoints 1, breakpoint 2, and breakpoint 3. Among them, the handwritten text trajectory corresponding to breakpoint 1, which can also be understood as the handwritten text trajectory between breakpoint 1 and breakpoint 2, is "ren", the handwritten text trajectory corresponding to breakpoint 2 is the part of the connected writing trajectory in "jin", and the handwritten text trajectory corresponding to breakpoint 3 is "er".

[0240] In step 820, from the recognized breakpoints, a first breakpoint is determined. The first breakpoint is the breakpoint closest to the current time point among the breakpoints whose disconnection time exceeds a first duration. The disconnection time is the time when a writing tool such as a finger or a stylus disconnects from the touch screen, that is, the time when there is no contact.

[0241] The first duration corresponds to the disconnection time. The first duration can be specifically set according to the disconnection time between two adjacent strokes in the historical handwritten database. Additionally, the writing habits of each object are different, so the first duration can change as the handwritten text prediction progresses to achieve the purpose of matching the terminal user object, realizing personalized setting of the first duration, thereby improving the accuracy of the handwritten text prediction method.

[0242] Refer to Figure 9, the multiple breakpoints obtained by breakpoint identification are arranged in chronological order as breakpoint 1, breakpoint 2, and breakpoint 3, wherein breakpoint 3 is the breakpoint closest to the current time point, and the disconnection time of breakpoint 3 exceeds the first duration, then for Figure 9 The first breakpoint corresponding to the input handwritten text trajectory shown is breakpoint 3.

[0243] In step 830, the input handwritten text trajectory starting from the first breakpoint is used as the target handwritten text trajectory. The target handwritten text trajectory is the input handwritten text trajectory between the first breakpoint and the current time point.

[0244] like Figure 9 As shown, breakpoint 3 is the first breakpoint, then Figure 9 The target handwritten text track in is the input handwritten text track “2” starting from breakpoint 3.

[0245] In the embodiment of steps 810 to 830 described above, the target handwritten text trajectory is determined by using a breakpoint recognition method. From among the multiple breakpoints corresponding to the handwritten text trajectory input after the start of handwritten text input, the breakpoint whose disconnection time exceeds the first duration and is closest to the current time point is determined as the first breakpoint, and the input handwritten text trajectory starting from the first breakpoint is used as the target handwritten text trajectory. The disclosed embodiment determines the target handwritten text trajectory based on breakpoint recognition, enabling precise division of individual strokes, thereby reducing the processing load of subsequent temporal-spatial feature extraction and improving the prediction efficiency of handwritten text.

[0246] Detailed description of step 320

[0247] In step 320 , the target handwritten text trajectory is sampled in time sequence to obtain time series sampling coordinates.

[0248] In one embodiment, the target handwritten text trajectory is formed by the movement of the finger on the touch screen. Figure 10 , step 320 includes:

[0249] Step 1010: Obtain the movement speed of the finger on the touch screen;

[0250] Step 1020: Determine the sampling frequency based on the moving speed;

[0251] Step 1030: Sample the target handwritten text trajectory based on the sampling frequency to obtain time series sampling coordinates.

[0252] It should be noted that in the handwritten text prediction application interface, users can write by moving their fingers on the touch screen to generate a target handwritten text trajectory. Alternatively, users can write on the touch screen using a stylus, capacitive pen, or other handwriting tool to generate a target handwritten text trajectory. The touch screen can specifically be the touch screen of a mobile phone, tablet, digitizer, or tablet.

[0253] Steps 1010 to 1030 are described in detail below.

[0254] In step 1010, the finger movement speed on the touch screen is obtained. If the target handwritten text trajectory is formed by the movement of a finger on the touch screen, the finger movement speed on the touch screen needs to be obtained. If the target handwritten text trajectory is formed by the movement of a handwriting tool such as a stylus or capacitive pen on the touch screen, the corresponding handwriting tool movement speed on the touch screen needs to be obtained.

[0255] In step 1020, a sampling frequency is determined based on the movement speed. The sampling frequency is the frequency of temporally up-sampling the target handwritten text trajectory.

[0256] Different objects move at different speeds when using fingers or handwriting tools to input handwriting on a touch screen. When the sampling frequency is low, each sampling does not obtain a coordinate point, but rather a stroke, or even the stroke corresponding to a word or phrase. When the sampling frequency is high, multiple samplings in the same time period may obtain the same coordinate point, resulting in a large number of repeated positions in the obtained time series sampling coordinates, increasing data redundancy. Therefore, it is necessary to set the sampling frequency based on the movement speed of the finger or handwriting tool on the touch screen. The movement speed is proportional to the sampling frequency. The faster the movement speed, the higher the sampling frequency. Conversely, the slower the movement speed, the lower the sampling frequency.

[0257] In step 1030, the target handwritten text trajectory is sampled based on the sampling frequency to obtain time-series sampling coordinates. The target handwritten text trajectory is sampled in time sequence based on the sampling frequency to obtain time-series sampling coordinates.

[0258] The sampling of the target handwritten text trajectory is determined by the sampling frequency, and the temporal arrangement of the sampled time-series coordinates is also determined by the sampling frequency. The sampling period is the reciprocal of the sampling frequency, and the interval between two temporally adjacent time-series coordinates is the reciprocal of the sampling frequency.

[0259] The embodiment of the above steps 1010 to 1030 determines the frequency of sampling the target handwritten text trajectory based on the movement speed of the finger on the touch screen, so that the sampling results obtained are coordinates rather than single or multiple strokes, and the time-series sampling coordinates correspond one-to-one to the sampling time, which improves the accuracy of the time-series sampling coordinates and reduces the data redundancy of the time-series sampling coordinates, thereby improving the efficiency of text prediction.

[0260] In one embodiment, the embodiment of the present disclosure can sample the target handwritten text trajectory in time sequence by random sampling. The random sampling method does not need to obtain the movement speed of the finger on the touch screen, which can improve the sampling speed of the target handwritten text trajectory.

[0261] In one embodiment, the sampling frequency corresponding to sampling the target handwritten text trajectory can be set to a fixed value. Specifically, the sampling frequency can be determined based on the average movement speed of all objects on the touch screen in the historical database, so that the sampling frequency can be suitable for most people while improving the sampling speed of the target handwritten text trajectory.

[0262] Detailed description of step 330

[0263] In step 330 , temporal spatial features of the target handwritten text trajectory are extracted from the temporal sampling coordinates.

[0264] In one embodiment, the time series sampling coordinates are the coordinates of multiple sampling points in time sequence, referring to Figure 11 , step 330 includes:

[0265] Step 1110: Acquire sampling point features of the target handwritten text trajectory based on the time-series sampling coordinates;

[0266] Step 1120: Input the sampling point features into the time series space feature extraction model to obtain the time series space features.

[0267] It should be noted that the temporal sampling coordinates include the coordinates of multiple sampling points, each of which is obtained by sampling the target handwritten text trajectory once. The temporal sampling coordinates do not need to include the sampling time corresponding to each sampling point; they only need to be arranged in chronological order. Specifically, the sampling time sequence of each sampling point can be represented by numbers.

[0268] Step 1110 and step 1120 are described in detail below.

[0269] In step 1110, sampling point features of the target handwritten text trajectory are obtained based on the time-series sampling coordinates. Sampling point features refer to the features of each sampling point corresponding to the target handwritten text trajectory, and include at least positional features and time-series features of the sampling points. In addition, sampling point features may also include stroke features, character features, and word features. For example, sampling point features can indicate the start and end of a stroke, or the start and end of a character or word.

[0270] In step 1120, the sampling point features are input into the temporal-spatial feature extraction model to obtain temporal-spatial features. The temporal-spatial feature model processes the sampling point features, converting them into temporal-spatial features to meet the processing requirements of the fusion prediction model. Temporal-spatial features represent the temporal and spatial features of the target handwritten text trajectory. They can be obtained by concatenating or fusing the temporal and spatial features.

[0271] Temporal-spatial feature extraction models can be encoder models, mapping models, neural network models, and so on. Sampling point features themselves can represent the temporal and spatial features of the target handwritten text trajectory. Therefore, temporal-spatial features can be obtained by encoding sampling point features using an encoder model or simply mapping them using a mapping model. Compared to encoder and mapping models, neural network models can perform comprehensive and accurate feature extraction of sampling point features. The resulting temporal-spatial features can more comprehensively and accurately represent the characteristics of the target handwritten text trajectory.

[0272] The above steps 1110 and 1120 first process the temporal sampling coordinates to obtain the sampling point features corresponding to the target handwritten text trajectory, and then input the sampling point features into the temporal space feature extraction model to obtain the temporal space features. The extraction of sampling point features and the setting of the temporal space feature extraction model can process the temporal sampling coordinates more accurately, so the temporal space features can more accurately represent the temporal features and spatial features of the target handwritten text trajectory.

[0273] The above is a general description of step 1110 and step 1120. The specific implementation process of step 1110 and step 1120 is described in detail below.

[0274] In step 1110 , sampling point features of the target handwritten text trajectory are obtained based on the time-series sampling coordinates.

[0275] In one embodiment, referring to Figure 12 , step 1110 includes:

[0276] Step 1210: Determine a normalized displacement difference of the sampling point based on the first coordinate of the sampling point and the second coordinate of the sampling point preceding the sampling point.

[0277] Step 1220: Obtain the pen lift and drop mark of the sampling point;

[0278] Step 1230: Obtain the stroke first sampling point identifier, where the stroke first sampling point identifier indicates whether the sampling point is the first sampling point in the stroke where the sampling point is located;

[0279] Step 1240: Generate sampling point features of the target handwritten text trajectory based on the normalized displacement difference, the pen lift and pen drop identifiers, and the stroke first sampling point identifier.

[0280] Steps 1210 to 1240 are described in detail below.

[0281] In step 1210, a normalized displacement difference of the sampling point is determined based on the first coordinate of the sampling point and the second coordinate of the sampling point immediately preceding the sampling point. The first coordinate represents the position information of the current sampling point, and the second coordinate represents the position information of the sampling point immediately preceding the sampling point. The position information may include the horizontal and vertical coordinates of the sampling point. In addition to rectangular coordinates, the position information may also be expressed in polar coordinates or natural coordinates.

[0282] The previous sampling point of a sampling point is the sampling point before the sampling point in the time sequence. For example, if the time information corresponding to the current sampling point is the t-th moment, then the previous sampling point is the sampling point at the t-1th moment.

[0283] Normalized displacement difference refers to the result obtained by normalizing the displacement difference between the current sampling point and the previous sampling point. Normalization is easier to calculate and obtain more generalized results, but it does not change the essence of displacement difference.

[0284] In step 1220, the pen-up / down flag of the sampling point is obtained. The pen-up / down flag is used to indicate the pen-up / down state of the target handwritten text trajectory. The pen-up / down flag has at least two values, one for indicating the pen-up state and the other for indicating the pen-down state of the target handwritten text trajectory.

[0285] For a stroke, the subject usually needs to lift the finger or handwriting tool off the touch screen at the last point, so the last stroke is the pen-up state, and the remaining points of the stroke require the handwriting or handwriting tool to always maintain contact with the touch screen, so the first point and the middle points of the stroke are the pen-down state.

[0286] In step 1230, the stroke first sampling point identifier is obtained, which indicates whether the sampling point is the first sampling point in the stroke in which the sampling point is located. The stroke first sampling point identifier can indicate whether the sampling point is the first point in the stroke in which the sampling point is located.

[0287] The stroke first sampling flag is set to at least two values, one of which is used to indicate that the sampling point is the first point in the stroke where it is located, and the other is used to indicate that the sampling point is not the first point in the stroke where it is located.

[0288] In step 1240, sampling point features of the target handwritten text trajectory are generated based on the normalized displacement difference, the pen lift and drop markers, and the stroke first sampling point marker. The normalized displacement difference, the pen lift and drop markers, and the stroke first sampling point marker are used as the sampling point features of the target handwritten text trajectory. The normalized displacement difference represents the positional features of the corresponding sampling point, while the pen lift and drop markers and the stroke first sampling point markers of multiple sampling points can confirm the start and end of the stroke.

[0289] In addition, the temporal characteristics of each sampling point can be determined based on the arrangement of the normalized displacement differential, pen lift and drop markers, and stroke start sampling point markers corresponding to each sampling point in the sampling point characteristics. Specifically, when the sampling point characteristics are expressed in the form of a matrix, the rows of the matrix correspond to the sampling points, and the multiple elements of a row of the matrix are the normalized displacement differential, pen lift and drop markers, and stroke start sampling point markers of the corresponding sampling point. The position of the row in the matrix can be used to represent the temporal characteristics of the sampling point. For example, the first row of the matrix corresponds to the sampling point at time 1, and the tth row of the matrix corresponds to the sampling point at time t.

[0290] It should be noted that the sampling point features only need to represent the spatial features and temporal features of the target handwritten text trajectory. Therefore, the sampling point features can be determined only based on the normalized displacement difference, or the sampling point features can also be the position coordinates of the sampling point and its corresponding previous sampling point.

[0291] Reference Figure 13 The normalized displacement difference of the sampling point is determined based on the position coordinates of the sampling point and its corresponding previous sampling point, and then the pen lift and pen drop identifier and the stroke first sampling point identifier are obtained, so as to generate the sampling point feature based on the normalized displacement difference, the pen lift and pen drop identifier, and the stroke first sampling point identifier.

[0292] The embodiment of steps 1210 to 1240 described above generates sampling point features based on the normalized displacement difference, the pen lift and drop markers, and the stroke first sampling point marker. The normalized displacement difference can represent the positional features of each sampling point corresponding to the target handwritten text trajectory, the pen lift and drop markers and the stroke first sampling point marker are used to represent the features of the stroke in which the sampling point is located, and based on the order of the normalized displacement difference, the pen lift and drop markers, and the stroke first sampling point markers corresponding to each sampling point, the temporal information of the sampling point can be determined. In addition to the spatial and temporal information of the target handwritten text trajectory, the sampling point features also include stroke information, thereby making the predicted text information more accurate.

[0293] The above is a general description of steps 1210 to 1240 . The specific implementation process of steps 1210 to 1230 will be described in detail below.

[0294] In step 1210 , a normalized displacement difference of the sampling point is determined based on the first coordinate of the sampling point and the second coordinate of the sampling point preceding the sampling point.

[0295] In one embodiment, the first coordinate includes a first horizontal coordinate and a second vertical coordinate, and the second coordinate includes a second horizontal coordinate and a second vertical coordinate. Figure 14 , step 1210 includes:

[0296] Step 1410: Determine a first difference between a first horizontal coordinate of the sampling point and a second horizontal coordinate of a sampling point preceding the sampling point, and a second difference between a first vertical coordinate of the sampling point and a second vertical coordinate of the sampling point preceding the sampling point.

[0297] Step 1420: Obtain the maximum value and minimum value of the vertical coordinate of the target handwritten text trajectory;

[0298] Step 1430: Determine the vertical coordinate span based on the vertical coordinate maximum value and the vertical coordinate minimum value;

[0299] Step 1440: Determine the normalized horizontal displacement differential of the sampling point based on the first difference and the vertical coordinate span, and determine the normalized vertical displacement differential of the sampling point based on the second difference and the vertical coordinate span;

[0300] Step 1450: Determine the normalized displacement difference of the sampling point based on the normalized horizontal coordinate displacement difference and the normalized vertical coordinate displacement difference.

[0301] It should be noted that compared with polar coordinates and natural coordinates, the calculation of rectangular coordinates is more convenient and faster. Therefore, the embodiment of the present disclosure uses rectangular coordinates to represent the coordinate positions of the sampling points and then calculates the normalized displacement difference.

[0302] The specific implementation process of steps 1410 to 1450 is described in detail below.

[0303] In step 1410, a first difference between a first abscissa of a sampling point and a second abscissa of a sampling point preceding the sampling point, and a second difference between a first ordinate of a sampling point and a second ordinate of a sampling point preceding the sampling point are determined. The first difference is the difference between the abscissas of the sampling point and its corresponding preceding sampling point, i.e., the difference between the first abscissa and the second abscissa. The second difference is the difference between the ordinates of the sampling point and its corresponding preceding sampling point, i.e., the difference between the first ordinate and the second ordinate.

[0304] The first difference can be expressed as x t -x t-1 , where x t is the horizontal coordinate of the sampling point at time t, that is, the first horizontal coordinate of the sampling point, x t-1 is the horizontal coordinate of the sampling point at time t-1, that is, the second horizontal coordinate of the previous sampling point. Correspondingly, the second difference is expressed as y t -y t-1 , where y t is the ordinate of the sampling point at time t, that is, the first ordinate of the sampling point, y t-1 is the ordinate of the sampling point at time t-1, that is, the second ordinate of the previous sampling point.

[0305] In step 1420, the maximum and minimum ordinate values ​​of the target handwritten text trajectory are obtained. The maximum ordinate value refers to the maximum value of the ordinates of the multiple sampling points corresponding to the target handwritten text trajectory, and the minimum ordinate value refers to the minimum value of the ordinates of the multiple sampling points corresponding to the target handwritten text trajectory.

[0306] In different application scenarios, the methods for determining the maximum and minimum values ​​of the vertical coordinate are different. Figure 2A If the object is inputting a handwriting text trajectory in the handwriting area of ​​the handwriting text prediction application, and the predicted text information is displayed in other text areas, then the maximum ordinate value refers to the maximum ordinate value of the sampling points corresponding to all handwriting text trajectories displayed in the handwriting area at the current time point, and the minimum ordinate value refers to the minimum ordinate value of the sampling points corresponding to all handwriting text trajectories displayed in the handwriting area at the current time point. For example, Figure 2A The maximum value of the vertical coordinate shown is the maximum value of the vertical coordinates of multiple sampling points corresponding to "day", and the minimum value of the vertical coordinate is the minimum value of the vertical coordinates of multiple sampling points corresponding to "day". Both the maximum value and the minimum value of the vertical coordinate are irrelevant to "today's weather". In addition, if the object is in the text area of ​​the handwritten text prediction application, such as Figure 2AIn the area where "Today's weather" is located, directly input the information. Then the maximum value of the vertical coordinate is the maximum value of the vertical coordinate of the input handwritten text trajectory corresponding to the row where the target handwritten text trajectory is located, and the minimum value of the vertical coordinate is the minimum value of the vertical coordinate of the input handwritten text trajectory corresponding to the row where the target handwritten text trajectory is located.

[0307] In step 1430, the vertical coordinate span is determined based on the vertical coordinate maximum and the vertical coordinate minimum. The vertical coordinate span refers to the span of the input text track corresponding to the target handwritten text track on the vertical coordinate. The vertical coordinate span can be specifically expressed as the difference between the vertical coordinate maximum and the vertical coordinate minimum, that is, y max -y min , where y max is the maximum value of the vertical coordinate, y min is the minimum value of the vertical axis.

[0308] In step 1440, a normalized horizontal displacement differential of the sampling point is determined based on the first difference and the vertical coordinate span, and a normalized vertical displacement differential of the sampling point is determined based on the second difference and the vertical coordinate span. The normalized vertical displacement differential refers to the result obtained by normalizing the displacement differential between the current sampling point and the previous sampling point in the vertical coordinate axis. The normalized horizontal displacement differential refers to the result obtained by normalizing the displacement differential between the current sampling point and the previous sampling point in the horizontal coordinate axis. Specifically, the embodiments of the present disclosure normalize the displacement differentials, i.e., the first difference and the second difference, by the vertical coordinate span.

[0309] The normalized horizontal coordinate displacement difference is determined based on the first difference and the vertical coordinate span. The normalized horizontal coordinate displacement difference can be expressed as:

[0310]

[0311] Where Δx t is the sampling point at time t, that is, the normalized horizontal coordinate displacement difference of the current sampling point, x t -x t-1 is the first difference, y max -y min is the vertical axis span.

[0312] The normalized ordinate displacement difference is determined based on the second difference and the ordinate span. The normalized ordinate displacement difference can be expressed as:

[0313]

[0314] Where Δy t is the sampling point at time t, that is, the normalized vertical coordinate displacement difference of the current sampling point, y t -y t-1 is the first difference, ymax -y min is the vertical axis span.

[0315] In step 1450, the normalized displacement differential of the sampling point is determined based on the normalized horizontal displacement differential and the normalized vertical displacement differential. In the embodiment of the present disclosure, the coordinates of the sampling point are expressed using rectangular coordinates, so the normalized displacement differential of the sampling point includes the normalized horizontal displacement differential and the normalized vertical displacement differential.

[0316] If the normalized horizontal displacement difference is Δx t , the normalized vertical coordinate displacement difference is Δy t , then the normalized displacement difference of the sampling point can be expressed as (Δx t ,Δy t ).

[0317] Reference Figure 15 , obtain the first ordinate and first abscissa of the current sampling point and the second ordinate and second abscissa of the corresponding previous sampling point. Based on the difference between the first abscissa and the second abscissa, obtain the first difference, and based on the difference between the first ordinate and the second ordinate, obtain the second difference. In addition, based on the input handwritten text trajectory corresponding to the target handwritten text trajectory, determine the ordinate maximum and the ordinate minimum, and use the difference between the ordinate maximum and the ordinate minimum as the ordinate span. Afterwards, normalize the first difference based on the ordinate span to obtain the normalized abscissa displacement difference, and normalize the second difference based on the ordinate span to obtain the normalized ordinate displacement difference. Finally, determine the normalized displacement difference of the sampling point based on the normalized abscissa displacement difference and the normalized ordinate displacement difference.

[0318] In the embodiment of steps 1410 through 1450, the first and second differences of the sampling points are normalized based on the vertical coordinate span of the target handwritten text trajectory to obtain normalized horizontal and vertical displacement differences, thereby determining the normalized displacement differences of the sampling points. Compared to directly using the sampling point coordinates or displacement differences as sampling point features, the normalized displacement differences obtained after normalization can reduce problems such as overfitting without losing displacement difference information, thereby making the handwritten text prediction process more stable.

[0319] In step 1220, the pen lift and drop identifiers of the sampling points are obtained.

[0320] In one embodiment, referring to Figure 16 , step 1220 includes:

[0321] Step 1610: Obtain a first pressure at a sampling point and a second pressure at a sampling point immediately before the sampling point;

[0322] Step 1620: If the first pressure is less than the second pressure, determine the pen-lifting / lowering flag as a first value indicating pen-lifting;

[0323] Step 1630: If the first pressure is not less than the second pressure, determine the pen-up / pen-down flag as the second value indicating pen-down.

[0324] Steps 1610 to 1630 are described in detail below.

[0325] In step 1610, a first pressure at a sampling point and a second pressure at a sampling point immediately preceding the sampling point are obtained. The first pressure refers to the pressure exerted on the touch screen of the terminal by a finger or writing tool when the sampling point is input. The second pressure refers to the pressure exerted on the touch screen of the terminal by a finger or writing tool when the sampling point corresponding to the sampling point is input immediately preceding the sampling point.

[0326] In step 1620, if the first pressure is less than the second pressure, the pen-lifting / lowering flag is determined to be a first value indicating pen-lifting. The first value is used to indicate that the pen-lifting / lowering state corresponding to the sampling point is the pen-lifting state.

[0327] When inputting a handwritten text trajectory, the subject needs to control their finger or writing tool to lift off the touch screen to complete a stroke or character. At this point, the sampling point of the handwritten text trajectory is in the pen-up state. In the pen-up state, the finger or writing tool is about to lift off the touch screen, and the pressure between it and the touch screen is generally less than at other times. Therefore, the first pressure at the sampling point in the pen-up state is less than the second pressure at the previous sampling point.

[0328] The first value can be set as needed, and the specific value of the first value only needs to be different from the value indicating the pen is put down. For ease of calculation, the embodiment of the present disclosure sets the first value to 0.

[0329] It should be noted that some subjects may still maintain the habit of writing on paper when inputting handwritten text trajectories. In this case, the pressure applied to the touch screen will decrease gradually during the final portion of a stroke, such as a left-falling stroke or a right-falling stroke. Based on this, embodiments of the present disclosure may be configured with a pressure threshold. When the first pressure is less than the pressure threshold, the pen-lifting / pen-dropping indicator is determined to be the first value indicating pen-lifting. The pressure threshold can be set based on the subject's writing habits.

[0330] In step 1630, if the first pressure is not less than the second pressure, the pen-up / down flag is determined to be a second value indicating pen-down. The second value is used to indicate that the pen-up / down state corresponding to the sampling point is the pen-down state.

[0331] When a subject enters the first point of a stroke, and the sampling point corresponding to the previous sampling point is the last point of the previous stroke, the first pressure is greater than the second pressure. However, when the subject is entering the middle of a stroke, the first pressure at the sampling point is typically equal to the second pressure at the previous sampling point. Therefore, if the first pressure is not less than the second pressure, the pen-up / pen-down flag is determined to be the second value indicating pen-down.

[0332] The second value can be set as needed, and the specific value of the second value only needs to be different from the value indicating the pen is lifted. For ease of calculation, the embodiment of the present disclosure sets the second value to 1.

[0333] If the first value is set to 0 and the second value is set to 1, the pen-lifting and pen-lowering flags of the first sampling point and the middle sampling point corresponding to the stroke are 1, and the pen-lifting and pen-lowering flag of the last sampling point corresponding to the stroke is 0.

[0334] It should be noted that, in the embodiment of the present disclosure, a pressure threshold may also be provided. When the first pressure is greater than or equal to the pressure threshold, the pen-lifting / pen-downing mark is determined as the second value indicating pen-down.

[0335] Reference Figure 17 First, the first pressure at the current sampling point and the second pressure at the corresponding previous sampling point are obtained. When the first pressure is less than the second pressure, the pen-up / down flag of the current sampling point is set to the first value, indicating that the sampling point is in the pen-up state. When the first pressure is not less than the second pressure, the pen-up / down flag of the current sampling point is set to the second value, indicating that the sampling point is in the pen-down state.

[0336] The embodiments of the above steps 1610 to 1630 determine the specific value of the pen lift and pen drop identifier based on the pressure of the sampling point and its corresponding previous sampling point, so that the obtained value of the pen lift and pen drop identifier is more accurate, thereby being able to better determine the stroke features based on the pen lift and pen drop identifier, thereby improving the accuracy of the predicted text information.

[0337] In step 1230, the identifier of the first sampling point of the stroke is obtained.

[0338] In one embodiment, referring to Figure 18 Before step 1210, the handwritten text prediction method provided by the embodiment of the present disclosure further includes:

[0339] Step 1810: Setting an interruption sampling point at a position where a stroke of the target handwritten text trajectory is interrupted, wherein the abscissa of the interruption sampling point is outside the abscissa value range of the target handwritten text trajectory, and the ordinate of the interruption sampling point is outside the ordinate value range of the target handwritten text trajectory;

[0340] Step 1820: adding the interrupted sampling point to the multiple sampling points;

[0341] Corresponding to step 1810 and step 1820, step 1230 includes:

[0342] Step 1830, if the previous sampling point of the sampling point is an interrupted sampling point, set the stroke start sampling point identifier to a third value;

[0343] Step 1840, if the previous sampling point of the sampling point is not an interrupted sampling point, set the stroke start sampling point identifier to a fourth value.

[0344] The following gives a detailed description of steps 1810 to 1840.

[0345] In step 1810, for the position of the stroke interruption of the target handwritten text trajectory, set an interrupted sampling point, where the abscissa of the interrupted sampling point is outside the abscissa value range of the target handwritten text trajectory, and the ordinate of the interrupted sampling point is outside the ordinate value range of the target handwritten text trajectory.

[0346] For sampling methods such as sampling at a fixed time interval and random sampling, the multiple sampling points obtained may not have the last point of the stroke, and it is impossible to determine whether each stroke is a newly input stroke. Based on this, in the embodiments of the present disclosure, an interrupted sampling point is set at the position of the stroke interruption of the target handwritten text trajectory.

[0347] An interrupted sampling point refers to the interruption position between two adjacent strokes. An interrupted sampling point usually represents the end of a stroke. For example, Figure 2A in the shown interface, the interrupted sampling point is the interruption position between adjacent strokes. For a handwritten text trajectory with connected strokes, an interrupted sampling point refers to the interruption position between two adjacent handwritten text trajectories. For example, Figure 9 as shown, one interrupted sampling point is the interruption position between the connected strokes of "人" and "今", and the other interrupted sampling point is the interruption position between "今" and "二".

[0348] It should be noted that there is no actual point of the interrupted sampling point in the already input handwritten text trajectory. Therefore, to facilitate the distinction between the interrupted sampling point and the sampling point corresponding to the already input handwritten text trajectory, in the embodiments of the present disclosure, the abscissa of the interrupted sampling point is determined outside the abscissa value range of the target handwritten text trajectory, and the ordinate of the interrupted sampling point is determined outside the ordinate value range of the target handwritten text trajectory. It can be said that the intersection of the value range of the coordinates of the interrupted sampling point and the value range of the coordinates of the sampling point corresponding to the target handwritten text trajectory is an empty set.

[0349] In step 1820, the interruption sampling point is added to the multiple sampling points. After the interruption sampling point is determined, the interruption sampling point is added to the multiple sampling points, and the interruption sampling point is located between the sampling points on both sides of its corresponding interruption position to facilitate the distinction of each stroke.

[0350] In step 1830, if the sampling point preceding the sampling point is an interrupt sampling point, the stroke first sampling point identifier is set to a third value. The third value is used to indicate that the sampling point is the first sampling point in the stroke in which it is located.

[0351] The interrupt sampling point is set at the stroke interruption position of the target handwritten text trajectory. The previous sampling point corresponding to the interrupt sampling point is the last sampling point in the stroke, and the next sampling point corresponding to the interrupt sampling point is the first sampling point in the stroke. Therefore, if the sampling point before the interrupt sampling point is the interrupt sampling point, then the sampling point becomes the first sampling point in the stroke.

[0352] In step 1840, if the sampling point preceding the sampling point is not an interrupt sampling point, the stroke first sampling point flag is set to a fourth value, which indicates that the sampling point is not the first sampling point in the stroke.

[0353] The interrupt sampling point is set at the position where the stroke of the target handwritten text trajectory is interrupted. The corresponding next sampling point is the first sampling point in the stroke. Therefore, if the sampling point before the sampling point is not an interrupt sampling point, the sampling point is not the first sampling point in the stroke.

[0354] It should be noted that the third value and the fourth value can be set to any value, as long as the third value and the fourth value are different from each other. In the embodiment of the present disclosure, the third value is set to 1 and the fourth value is set to 0.

[0355] In the embodiment of the above-mentioned steps 1810 to 1840, interruption sampling points are set at the positions where the strokes of the target handwritten text trajectory are interrupted, and the value of the stroke first sampling point flag is determined based on the interruption sampling points. Therefore, no matter what method is used to sample the target handwritten text trajectory, the multiple sampling points containing the interruption sampling points can be used to divide the multiple strokes corresponding to the target handwritten text trajectory, thereby obtaining predicted text information.

[0356] In step 1240, based on the normalized displacement difference, the pen lift and drop mark, and the stroke first sampling point mark, the sampling point feature of the target handwritten text trajectory is generated. The sampling point feature of the target handwritten text trajectory is a four-dimensional feature, which can be specifically represented by (Δx t ,Δy t ,p t ,n t ), where (Δx t,Δy t ) is the normalized displacement difference of the sampling point, p t For pen lift and drop marks, n t is the first sampling point identifier of the stroke, and t represents the sampling point at the tth moment. For ease of calculation, the embodiment of the present disclosure defines Δx0=0, Δy0=0. The first value of the pen-lifting and pen-lowering identifier, that is, the value corresponding to the pen-lifting state is 0, the second value of the pen-lifting and pen-lowering state identifier, that is, the value corresponding to the pen-lowering state is 1, and the third value of the stroke first sampling point identifier, that is, the sampling point is the first sampling point of the stroke where it is located, is 1. Otherwise, the stroke first sampling point is equal to the fourth value 0. Then the first sampling point corresponding to the target text trajectory where the stroke is located corresponds to (p t ,n t ) is expressed as (first value, third value), that is, (1,1). The sampling point in the middle of the stroke corresponds to (p t ,n t ) is represented as (first value, fourth value), i.e. (1,0). The last sampling point of the stroke corresponds to (p t ,n t ) is represented as (first value, fourth value), that is, (0,0).

[0357] In addition, the embodiment of the present disclosure sets an interruption sampling point at the position where the stroke of the target handwritten text trajectory is interrupted. The pressure between the finger and the touch screen corresponding to the interruption sampling point is 0. In order to distinguish it from other sampling points of the stroke, the embodiment of the present disclosure sets the pen lift and drop mark corresponding to the interruption sampling point to the fifth value. The fifth value is different from the first value and the second value. If the first value is set to 0 and the second value is set to 1, then the fifth value can be set to -1. The (p t ,n t ) can be expressed as (-1,0). At the interrupt sampling point corresponding to (Δx t ,Δy t ,p t ,n t ), Δx t , Δy t The coordinate value of the target handwritten text trajectory is outside the range, p t is -1, n t In fact, the interrupt sampling point can be identified according to the value of the pen lift and drop flag, so the stroke first sampling point flag of the interrupt sampling point can also be other values, such as 1.

[0358] In step 1120, the sampling point features are input into the time series space feature extraction model to obtain the time series space features.

[0359] In one embodiment, the temporal spatial feature extraction model includes a long short-term memory network and a convolutional layer, referring to Figure 19 , step 1120 includes:

[0360] Step 1910: Input the sampling point features into the long short-term memory network to obtain a first intermediate vector;

[0361] Step 1920: Input the first intermediate vector into the convolution layer to obtain temporal spatial features.

[0362] Step 1910 and step 1920 are described in detail below.

[0363] In step 1910, the sampling point features are input into a long short-term memory network to obtain a first intermediate vector. The long short-term memory network is a temporal recurrent network that can extract long-term and short-term information from sampling point features, thereby effectively extracting the temporal features of the target handwritten text trajectory.

[0364] The first intermediate vector is the output of the long short-term memory network processing the sampling point features. The first intermediate vector contains the temporal and spatial features of the target handwritten text trajectory. The elements in the first intermediate vector can be evenly divided based on the number of sampling points, and the average division result corresponds to the sampling point features of each sampling point.

[0365] The first intermediate vector is (!@A1, !@B1, !@C1, !@D1, ..., !@At, !@Bt, !@Ct, !@Dt), where !@A1 is the normalized horizontal displacement difference of the sampling point at time 1, !@B1 is the normalized vertical displacement difference of the sampling point at time 1, !@C1 is the pen lift / pen drop identifier of the sampling point at time 1, and !@D1 is the marker of the first sampling point of the stroke at time 1. !@At is the normalized horizontal displacement difference of the sampling point at time t, !@Bt is the normalized vertical displacement difference of the sampling point at time t, !@Ct is the pen lift / pen drop identifier of the sampling point at time t, and !@Dt is the marker of the first sampling point of the stroke at time t.

[0366] In step 1920, the first intermediate vector is input into a convolutional layer to obtain a temporal spatial feature. The convolutional layer can perform secondary feature extraction on the first intermediate vector to improve the accuracy of the temporal spatial feature.

[0367] A convolutional layer consists of several convolutional units, and the parameters of each convolutional unit are optimized during training using the backpropagation algorithm. The goal of the convolution operation is to extract different features of the input, that is, to extract the temporal and spatial features from the first intermediate vector to obtain temporal and spatial features.

[0368] It should be noted that the convolution layer is set to further process the first intermediate vector, thereby improving the accuracy of the temporal spatial features. Therefore, the temporal spatial features in the embodiment of the present disclosure can only be set with a long short-term memory network.

[0369] Multiple long short-term memory networks and convolutional layers can be set to extract the temporal features and spatial features in the sampling point features multiple times, thereby improving the extraction accuracy of the temporal spatial feature extraction model for the temporal spatial features.

[0370] like Figure 4 The handwritten text prediction method shown uses a temporal-spatial feature extraction model to extract features from sampling points, generating temporal-spatial features. Specifically, the temporal-spatial feature extraction model includes a long short-term memory (LSTM) network and a convolutional layer. The sampling point features are first input into the LSTM network to generate a first intermediate vector. This first intermediate vector is then processed by the convolutional layer to generate the temporal-spatial features.

[0371] In the embodiments of steps 1910 and 1920, the temporal-spatial feature extraction model includes a long short-term memory (LSTM) network and a convolutional layer. The sampling point features are sequentially processed by the LSTM network and the convolutional layer to obtain temporal-spatial features. The LSTM network can initially extract the temporal and spatial features of the target handwritten text trajectory to obtain a first intermediate vector, while the convolutional layer processing can improve the accuracy of the temporal-spatial features, thereby improving the accuracy of the predicted text information.

[0372] The above is a general description of step 1910 and step 1920. The specific implementation process of step 1920 is described in detail below.

[0373] In step 1920, the first intermediate vector is input into the convolution layer to obtain temporal spatial features.

[0374] In one embodiment, the temporal spatial feature extraction model further includes a first linear layer and a maximum pooling layer, referring to Figure 20 , step 1920 includes:

[0375] Step 2010: Input the first intermediate vector into the first linear layer to obtain a second intermediate vector;

[0376] Step 2020: Input the second intermediate vector into the convolution layer to obtain a third intermediate vector;

[0377] Step 2030: Input the third intermediate vector into the maximum pooling layer to obtain temporal spatial features.

[0378] Steps 2010 to 2030 are described in detail below.

[0379] In step 210, the first intermediate vector is input into a first linear layer to obtain a second intermediate vector. The input of the first linear layer is connected to the output of the long short-term memory network, and the output of the first linear layer is connected to the input of the convolutional layer. The second intermediate vector is a vector result obtained by linearly combining or linearly transforming the first intermediate vector by the first linear layer.

[0380] Each neuron in the first linear layer is connected to all neurons in the long short-term memory network. The first linear layer mainly performs linear combination or linear transformation on the first intermediate vector, thereby integrating the local information corresponding to the spatial features and temporal features in the first intermediate vector.

[0381] In step 220, the second intermediate vector is input into a convolutional layer to obtain a third intermediate vector. The input of the convolutional layer is connected to the output of the first linear layer. The convolutional layer further extracts features from the second intermediate vector to improve the accuracy of the temporal spatial features obtained by the temporal spatial feature model. The third intermediate vector is the vector result obtained by convolution of the second intermediate vector by the convolutional layer.

[0382] In step 2030, the third intermediate vector is input into the maximum pooling layer to obtain the temporal spatial feature. The input end of the maximum pooling layer is connected to the output end of the convolution layer.

[0383] The max pooling layer downsamples the third intermediate vector, reducing the feature scale. This improves the convergence speed of the temporal-spatial feature extraction model, reduces computational effort, and lowers processing overhead. Furthermore, the max pooling layer reduces the dimensionality of the third intermediate vector and removes unnecessary features, retaining the key features, thereby improving the accuracy of temporal-spatial features.

[0384] Reference Figure 4 The temporal spatial feature extraction model provided by the embodiment of the present disclosure includes a long short-term memory network, a first linear layer, a convolutional layer, and a maximum pooling layer in sequence. The long short-term memory network is used to extract the long and short-term information in the sampling point features to obtain a first intermediate vector. The first linear layer is used to perform a linear combination or linear transformation on the first intermediate vector output by the long short-term memory network to integrate the local information in the first intermediate vector to obtain a second intermediate vector. The second intermediate vector is input into the convolutional layer for convolution calculation to obtain a third intermediate vector. Finally, the third intermediate vector is subjected to maximum pooling processing through the maximum pooling layer to obtain the temporal spatial feature.

[0385] The temporal-spatial feature extraction model in the embodiment of the above steps 2010 to 2030 includes a long short-term memory network, a first linear layer, a convolutional layer and a maximum pooling layer. The first linear layer can integrate local information in the first intermediate vector, and the maximum pooling layer can perform maximum pooling processing on the third intermediate vector output by the convolutional layer, thereby removing unnecessary information. The obtained temporal-spatial features contain more effective information and have better accuracy, thereby enabling high-accuracy handwritten text prediction.

[0386] Detailed description of step 350

[0387] In step 350, the temporal-spatial features and the above-mentioned information are input into the fusion prediction model to obtain the predicted text information.

[0388] In one embodiment, referring to Figure 21 , step 350 includes:

[0389] Step 2110, determining the embedding vector corresponding to the above-mentioned information;

[0390] Step 2120, adding the position encoding information of each character of the above-mentioned information to the embedding vector;

[0391] Step 2130, inputting the embedding vector and the temporal-spatial features into the fusion prediction model to obtain the predicted text information.

[0392] The following will describe steps 2110 to step 2130 in detail.

[0393] In step 2110, the embedding vector corresponding to the above-mentioned information is determined. The embedding vector refers to a vector that can represent the above-mentioned information.

[0394] Determining the embedding vector corresponding to the above-mentioned information is actually a process of converting the text information corresponding to the above-mentioned information into a vector, and there are various implementation methods for this process, such as encoders, mapping models, large language models, etc. Specifically, the above-mentioned information is encoded by an encoder, so as to convert the above-mentioned information into a vector form for expression, and the embedding vector is obtained. Inputting the above-mentioned information into the mapping model, the mapping result is the embedding vector. The large language model can also convert the above-mentioned information into a vector form.

[0395] When the above-mentioned information only includes a single character, word, sentence, or paragraph, and the above-mentioned information consists of multiple characters, then the multiple elements in the embedding vector correspond to the characters in the above-mentioned information. In addition, the above-mentioned information may also include strokes, and then the embedding vector is also provided with elements corresponding one-to-one to the strokes in the above-mentioned information.

[0396] In step 2120, the position encoding information of each character of the above-mentioned information is added to the embedding vector. The position encoding information refers to the encoding result obtained by encoding the position information of each character of the above-mentioned information.

[0397] Through the position encoding information, the specific position of each character in the above-mentioned information in the above-mentioned information can be determined. Specifically, when the above-mentioned information is "Today's weather", then the position information corresponding to the character "今" in this above-mentioned information is 1, the position information of the first character "天" is 2, the position information of the second character "天" is 3, the position information of the character "气" is 4, and then the position information is encoded to obtain the position encoding information. In the position encoding information, the position information corresponds one-to-one to each character in the above-mentioned information.

[0398] There are many ways to incorporate positional encoding information into an embedding vector. For example, an embedding vector can be expressed as (#$1,&P1,…,#$N,&PN), where “#$1” represents the first character in the text, “&P1” represents the encoding of the first character’s position, “#$1” represents the Nth character in the text, and “&PN” represents the position of the Nth character. Alternatively, an embedding vector can be expressed as (#$1,…,#$N,*&%,&P1,…,&PN), where “*&%” separates characters and represents the encoding corresponding to their positional information.

[0399] In step 2130, the embedding vector and the temporal-spatial features are input into a fusion prediction model to obtain predicted text information. A fusion prediction model is a model that can predict the text information to be input based on the embedding vector and the temporal-spatial features.

[0400] Reference Figure 4 The input of the fusion prediction model is the temporal spatial features of the target handwritten text trajectory and the context information. The context information needs to be further processed before it can be input into the fusion prediction model. First, the embedding vector corresponding to the context information is determined. Then, based on the position of each character in the context information, its corresponding position encoding information is determined. The position encoding information is concatenated with the embedding vector, or the position encoding information is inserted into the position of the corresponding character in the embedding vector to obtain a new embedding vector. This embedding vector and the temporal spatial features are used together as the input of the fusion prediction model.

[0401] The embodiment of steps 2110 to 2130 processes the context information to obtain an embedding vector after adding the position coding information, which serves as the input of the fusion prediction model. The embodiment of the present disclosure determines the embedding vector based on each character in the embedding vector and its corresponding position coding information, making the input of the fusion prediction model, i.e., the context information of the target handwritten text trajectory, clearer. As a result, the predicted text information obtained by the fusion prediction model based on the context information and temporal-spatial features is more accurate.

[0402] The above is a general description of steps 2110 to 2130. The specific implementation process of step 2130 is described in detail below.

[0403] In step 2130, the embedding vector and the temporal spatial features are input into the fusion prediction model to obtain the predicted text information.

[0404] In one embodiment, the fusion prediction model includes a multi-head attention model and a fusion layer, referring to Figure 22 , step 2130 includes:

[0405] Step 2210: Input the embedding vector into the multi-head attention model to obtain the attention vector;

[0406] Step 2220: Input the attention vector and temporal spatial features into the fusion layer to obtain a fusion vector;

[0407] Step 2230: Input the fusion vector into the decision layer to obtain the predicted text information.

[0408] Steps 2210 to 2230 are described in detail below.

[0409] In step 2210, the embedding vector is input into a multi-head attention model to obtain an attention vector. A multi-head attention model is a model that can effectively extract information from the embedding vector to obtain an attention vector. The attention vector is the result of processing the embedding vector based on the multi-head attention model.

[0410] The Multi-Head Attention model can focus on and process text information in the embedding vector from multiple perspectives, allowing the fusion prediction model to learn more diverse and complex features. Specifically, the Multi-Head Attention model uses multiple sets of attention weights, each of which can learn different semantic information from the embedding vector and generate a context vector. Finally, these context vectors are spliced ​​together to obtain the attention vector.

[0411] In step 2220, the attention vector and the temporal spatial features are input into the fusion layer to obtain a fusion vector. The fusion layer refers to a network layer that can fuse the attention vector and the temporal spatial features. The fusion vector is the result vector obtained by fusing the attention vector and the temporal spatial features through the fusion layer.

[0412] The fusion layer can fuse the attention vector and temporal spatial features by simple cascading, or by Hadamard product, linear fusion, nonlinear fusion and other fusion methods.

[0413] In step 2230, the fused vector is input into the decision layer to obtain the predicted text information. The decision layer can convert the fused vector into a prediction probability and determine the text information with the highest prediction probability as the predicted text information.

[0414] The decision layer can use a variety of different activation functions to process the fused vector to convert the fused vector into a predicted probability, such as the softmax function, ReLU function, sigmoid function, etc. In the embodiment of the present disclosure, the softmax function is specifically used to process the fused vector. Compared with other activation functions, softmax can simultaneously output the probability value of each piece of text information without a binary classification model, making it easier to obtain predicted text information.

[0415] Reference Figure 4 The fusion prediction model mainly consists of a multi-head attention model, a fusion layer, and a decision layer. The multi-head attention model processes the embedding vector corresponding to the context information to obtain an attention vector. The temporal spatial features are fused with the attention vector to obtain a fusion vector, which can be used to represent the temporal features, spatial features, and context information of the target handwritten text trajectory. The decision layer then converts the fusion vector into multiple prediction probabilities corresponding to the text information, and the text information with the highest prediction probability is determined as the predicted text information.

[0416] In the embodiment of steps 2210 to 2230 above, the fusion prediction model sequentially includes a multi-head attention model, a fusion layer, and a decision layer. This fusion prediction model integrates the temporal and spatial features of the target handwritten text trajectory with the contextual information to produce predicted text information. Furthermore, compared to other self-attention mechanisms, the multi-head attention model improves the accuracy of handwritten text prediction.

[0417] The above is a general description of steps 2210 to 2230. The specific implementation process of steps 2220 and 2230 will be described in detail below.

[0418] In step 2220, the attention vector and the temporal-spatial features are input into the fusion layer to obtain a fusion vector.

[0419] In one embodiment, referring to Figure 23 , step 2220 includes:

[0420] Step 2310: Superimpose the attention vector and the embedding vector to obtain a first superimposed vector;

[0421] Step 2320: normalize the first superposition vector to obtain a first normalized vector;

[0422] Step 2330: Input the first normalized vector and the temporal spatial feature into the fusion layer to obtain a fusion vector.

[0423] Steps 2310 to 2330 are described in detail below.

[0424] In step 2310, the attention vector is superimposed on the embedding vector to obtain a first superimposed vector. The first superimposed vector is the superimposed result of the attention vector and the embedding vector.

[0425] It should be noted that the disclosed embodiments superimpose the attention vector and the embedding vector using a residual connection. For example, if the embedding vector is x and the attention vector is f(x), then the first superimposed vector can be expressed as x + f(x). When backpropagating the fusion prediction model to update model parameters, the gradient vanishing problem is easily caused. However, the residual connection ensures that the derivative of the first superimposed vector includes the constant term "1", effectively solving this problem.

[0426] In step 2320, the first superimposed vector is normalized to obtain a first normalized vector. The first normalized vector is the normalized result of the first superimposed vector.

[0427] The disclosed embodiment can use layer normalization (LN), batch normalization, etc. to normalize the first superposition vector. Among them, layer normalization is a normalization operation performed on all neurons in a certain layer. It takes into account the stability of data distribution on the entire training set and can effectively alleviate the phenomenon of gradient explosion or disappearance. Batch normalization is a normalization operation performed on a single neuron in a neural layer in each training batch of the model. Its purpose is to ensure that the data of each training batch has the same distribution, thereby avoiding overfitting due to changes in data distribution.

[0428] In step 2330, the first normalized vector and the temporal spatial feature are input into a fusion layer to obtain a fused vector. The fused vector is specifically a fusion result of the first normalized vector and the temporal spatial feature.

[0429] Reference Figure 4 In this embodiment, a superposition and normalization model is implemented between the multi-head attention model. This model first superimposes the embedding vector and the attention vector to obtain a first superposition vector. This first superposition vector is then normalized to obtain a first normalized vector. This first normalized vector is then used as input to the fusion layer to generate a fused vector based on the first normalized vector and the temporal-spatial features.

[0430] In the embodiment of steps 2310 to 2330 described above, the attention vector and the embedding vector are first superimposed, and the resulting first superimposed vector is normalized to obtain a first normalized vector. The first normalized vector corresponds to the contextual information of the target handwritten text trajectory. This first normalized vector, along with the temporal-spatial features, is input into the fusion layer to obtain a fused vector. The superposition process effectively addresses the vanishing gradient problem, while normalization ensures the stability of the first normalized vector input into the fusion layer, thereby ensuring orderly handwritten text prediction.

[0431] The above is a general description of steps 2310 to 2330. The specific implementation process of step 2330 is described in detail below.

[0432] In step 2330, the first normalized vector and the temporal spatial feature are input into the fusion layer to obtain a fusion vector.

[0433] In one embodiment, referring to Figure 24 , step 2330 includes:

[0434] Step 2410: Input the first normalized vector and the temporal spatial feature into the fusion layer to obtain a fourth intermediate vector;

[0435] Step 2420: Superimpose the fourth intermediate vector, the first normalized vector, and the temporal spatial feature to obtain a second superimposed vector;

[0436] Step 2430: normalize the second superposition vector to obtain a second normalized vector;

[0437] Step 2440: Input the second normalized vector into the second linear layer to obtain a fifth intermediate vector;

[0438] Step 2450: Superimpose the second normalized vector and the fifth intermediate vector to obtain a third superimposed vector;

[0439] Step 2460: Normalize the third superposition vector to obtain a fusion vector.

[0440] Steps 2410 to 2460 are described in detail below.

[0441] In step 2410, the first normalized vector and the temporal spatial feature are input into a fusion layer to obtain a fourth intermediate vector. The fourth intermediate vector is a fusion result of the first normalized vector and the temporal spatial feature.

[0442] To ensure the accuracy of the predicted text information, multiple network layers are set between the fusion layer and the decision layer. Then, the first normalized vector and the temporal spatial feature are input into the fusion layer, and the direct result is the fourth intermediate vector.

[0443] In step 2420, the fourth intermediate vector, the first normalized vector, and the time-series spatial feature are superimposed to obtain a second superimposed vector. The second superimposed vector is the superimposed result of the fourth intermediate vector, the first normalized vector, and the time-series spatial feature.

[0444] The input of the fusion layer is the first normalized vector and the time-series spatial features, and the output is the fourth intermediate vector. Therefore, the fourth intermediate vector, the first normalized vector, and the time-series spatial features need to be superimposed. The embodiment of the present disclosure uses a residual connection to superimpose the fourth intermediate vector, the first normalized vector, and the time-series spatial features. For example, if the first normalized vector is m, the time-series spatial features are n, and the fourth intermediate vector is f(m+n), then the second superimposed vector can be expressed as m+n+f(m+n). Residual connections can effectively solve the problem of vanishing gradients.

[0445] In step 2430, the second superimposed vector is normalized to obtain a second normalized vector. The second normalized vector is the normalized result of the second superimposed vector.

[0446] It should be noted that the embodiment of the present disclosure may use layer normalization, batch normalization, or the like to perform normalization processing on the second superposition vector.

[0447] In step 2440, the second normalized vector is input into the second linear layer to obtain a fifth intermediate vector. The fifth intermediate vector is a vector result obtained by linearly combining or linearly transforming the second normalized vector by the second linear layer.

[0448] Each neuron in the second linear layer is connected to all neurons in the network layer that is normalized in step 2430. The second linear layer mainly performs linear combination or linear transformation on the second normalized vector, thereby integrating the spatial features, temporal features, and local information corresponding to the above information in the second normalized vector.

[0449] In step 2450, the second normalized vector and the fifth intermediate vector are superimposed to obtain a third superimposed vector. The third superimposed vector is the superimposed result of the second normalized vector and the fifth intermediate vector.

[0450] It should be noted that, in the embodiment of the present disclosure, the second normalized vector and the fifth intermediate vector are superimposed by a residual connection. The superposition method of the residual connection can reduce the frequency of gradient disappearance.

[0451] In step 2460, the third superimposed vector is normalized to obtain a fused vector. The fused vector is the normalized result of the third superimposed vector.

[0452] It should be noted that the embodiment of the present disclosure may use layer normalization, batch normalization, or the like to perform normalization processing on the third superposition vector.

[0453] Reference Figure 4In the disclosed embodiment, a superposition and normalization model, a second linear layer, and a superposition and normalization model are sequentially arranged between the fusion layer and the decision layer. The fusion layer fuses the first normalized vector and the time-series spatial features, resulting in a fourth intermediate vector. The output and input of the fusion layer, namely the fourth intermediate vector, the first normalized vector, and the time-series spatial features, are then superimposed and normalized to obtain a second normalized vector. The second normalized vector is linearly processed by the second linear layer to obtain a fifth intermediate vector. The input and output of the second linear layer, namely the second normalized vector and the fifth intermediate vector, are then superimposed to obtain a fused vector.

[0454] The embodiment of steps 2410 to 2460 performs multiple stacking, normalization, and linear processing on the output of the fusion layer, thereby increasing the effective information contained in the fusion vector input to the decision layer. Furthermore, the linear processing can integrate the spatial features, temporal features, and local information corresponding to the above information in the second normalized vector, while the stacking and normalization processing can reduce the vanishing gradient problem of the fusion prediction model during backpropagation training.

[0455] In step 2230, the fusion vector is input into the decision layer to obtain the predicted text information.

[0456] In one embodiment, referring to Figure 25 , step 2230 includes:

[0457] Step 2510: Input the fused vector into the third linear layer to obtain a sixth intermediate vector;

[0458] Step 2520: Input the sixth intermediate vector into the decision layer to obtain the probabilities of multiple candidate text information;

[0459] Step 2530: Determine the predicted text information based on the probabilities of the multiple candidate text information.

[0460] Steps 2510 to 2530 are described in detail below.

[0461] In step 2510, the fused vector is input into the third linear layer to obtain a sixth intermediate vector. The sixth intermediate vector is a result of pre-processing the fused vector based on the third linear layer.

[0462] Each neuron in the third linear layer is connected to all neurons in the network layer that is normalized in step 2460. The third linear layer mainly performs linear combination or linear transformation on the fusion vector, thereby integrating local information in the fusion vector.

[0463] In step 2520, the sixth intermediate vector is input into the decision layer to obtain the probabilities of multiple candidate text information. The candidate text information is multiple possible text information to be input after the target handwritten text trajectory. The probability of the candidate text information is the probability of the candidate text information being input after the target handwritten text trajectory. For example, when the input text trajectory is "Jin Yi", then the candidate text information can be "Tian" and "Ri", and the probability of "Tian" is greater than that of "Ri".

[0464] In step 2530, based on the probabilities of multiple candidate text information, the predicted text information is determined. The predicted text information is the candidate text information with the highest probability among multiple candidate text information. For example, when the input text trajectory is "Jin Yi", then the candidate text information can be "Tian" and "Ri", and the probability of "Tian" is greater than that of "Ri", then the predicted text information is "Tian".

[0465] It should be noted that in the embodiments of the present disclosure, multiple candidate text information can also be arranged in descending order according to the probability, and the first preset number of candidate text information can be used as the predicted text information, or multiple candidate text information with probabilities greater than the preset probability threshold can be used as the predicted text information. Then, multiple predicted text information is arranged and displayed on the touch screen in descending order of probability for the object to determine the text information expected to be input among the multiple displayed text information.

[0466] Refer to Figure 4 , a third linear layer is further provided between the decision layer and the fusion layer. The third linear layer is used to perform a linear combination or linear transformation on the fusion vector to obtain the sixth intermediate vector. The decision layer converts the sixth intermediate vector into the probabilities of multiple candidate text information, and uses the candidate text information with the highest probability as the predicted text information.

[0467] The embodiments of the above steps 2510 to 2530 are provided with a third linear layer to further integrate the fusion vector. The decision layer converts the sixth intermediate vector into the probabilities of multiple candidate text information, and determines the predicted text information based on the probabilities of multiple candidate text information. The setting of the third linear layer and the method of determining the predicted text information by probability can make the obtained predicted text information closer to the needs of the object and improve the accuracy of handwritten text prediction.

[0468] Joint training of the temporal-spatial feature extraction model and the integration prediction model

[0469] In an embodiment, refer to Figure 26 , the joint training method of the temporal-spatial feature extraction model and the integration prediction model includes the following steps:

[0470] Step 2610: Obtain a set of sample handwritten text trajectories, wherein the sample handwritten text trajectories in the set of sample handwritten text trajectories have a sample previous context label and a sample next context label;

[0471] Step 2620: Sample the sample handwritten text trajectory in time sequence to obtain sample time series sampling coordinates;

[0472] Step 2630: extracting the sample temporal spatial features of the sample handwritten text trajectory from the sample temporal sampling coordinates;

[0473] Step 2640: Input the sample temporal spatial features and the sample context labels into the fusion prediction model to obtain the predicted sample context;

[0474] Step 2650: Calculate the loss function based on the difference between the predicted sample context and the sample context label, and jointly train the temporal spatial feature extraction model and the fusion prediction model through the loss function.

[0475] The following is an overview of steps 2610 to 2650.

[0476] In step 2610, a set of sample handwritten text trajectories is obtained, wherein each sample handwritten text trajectory in the set has a sample context label and a sample context label. The sample handwritten text trajectories serve as training samples for the temporal spatial feature extraction model and the prediction model. The set of sample handwritten text trajectories is a collection of multiple sample handwritten text trajectories and their corresponding sample context labels and sample context labels.

[0477] The sample context label refers to the actual context information corresponding to the sample handwritten text trajectory, and the sample context label refers to the actual context information corresponding to the sample handwritten text trajectory. The sample context label corresponding to one sample handwritten text trajectory may be the sample context label of another sample handwritten text trajectory, and the sample context label corresponding to one sample handwritten text trajectory may be the sample context label of another sample handwritten text trajectory.

[0478] In step 2620, the sample handwritten text trajectory is sampled in time sequence to obtain sample time-series sampling coordinates. Sample time-series sampling coordinates refer to the coordinates of multiple points in the sample handwritten text trajectory that are consecutive in time sequence. The sample time-series sampling coordinates contain not only the position information of each point, but also the time-series information of each point. The time-series information can reflect the order of multiple strokes corresponding to the sample handwritten text trajectory and the direction of each stroke.

[0479] To sample the handwritten text trajectory in chronological order, we need to acquire new handwritten text trajectory points on the terminal at regular intervals as the sample handwritten text trajectory is input. This allows us to represent the sample handwritten text trajectory using the coordinates of multiple handwritten text trajectory points arranged in chronological order, i.e., the temporal sampling coordinates. In other words, the acquisition and sampling of the sample handwritten text trajectory are performed simultaneously.

[0480] Different terminal devices have different screen sizes and pixel counts. Therefore, the coordinate axes corresponding to the sample timing sampling coordinates need to be determined based on the terminal device, thereby determining the sample timing sampling coordinates corresponding to each sampling point in the sample handwritten text trajectory. In addition, the coordinate axes corresponding to different terminal devices can be converted to the same coordinate axis specifications, thereby converting the handwritten text trajectories input from different terminal devices to the same coordinate axis specifications for processing.

[0481] In step 2630, the sample temporal spatial features of the sample handwritten text trajectory are extracted from the sample temporal sampling coordinates. The sample temporal spatial features are the feature extraction results of the sample temporal sampling coordinates, which include the spatial features and temporal features of the sample handwritten text trajectory. The sample temporal features can be represented in the form of a matrix or vector.

[0482] In addition, in addition to the spatial features and temporal features of the sample handwritten text trajectory, the sample temporal spatial features may also include other content, such as the start mark and end mark of each stroke, so as to divide the strokes corresponding to the sample handwritten text trajectory, thereby being able to determine the multiple strokes corresponding to the sample handwritten text trajectory, thereby making the temporal spatial feature extraction model and the model integrated into the prediction model perform better. The method for extracting the sample temporal spatial features needs to be consistent with the method for extracting the temporal spatial features in step S330. Specifically, the embodiment of the present disclosure obtains the temporal spatial features through the temporal spatial feature extraction model.

[0483] In step 2640, the sample temporal spatial features and the sample context labels are input into the fusion prediction model to obtain a predicted sample context. The predicted sample context refers to the predicted sample context corresponding to the sample handwritten text trajectory obtained by processing the sample temporal spatial features and the sample context labels through the fusion prediction model.

[0484] In step 2650, a loss function is calculated based on the difference between the predicted sample context and the sample context label, and the temporal spatial feature extraction model and the fusion prediction model are jointly trained using the loss function. The loss function is determined based on the difference between the predicted sample context and the sample context label. After determining the loss function, the parameters of the temporal spatial feature extraction model and the fusion prediction model can be modified through backpropagation to reduce the difference between the predicted sample context and the sample context label. In this way, the predicted sample context obtained by the trained temporal spatial feature extraction model and the fusion prediction model is the same as the sample context label.

[0485] The loss function measures the distance between the predicted sample context and the expected value, i.e., the sample context label, obtained by the temporal spatial feature extraction model and the fusion prediction model. The loss function facilitates adjustment and control of model parameters. The disclosed embodiments may employ loss functions such as mean square error and cross entropy error.

[0486] Reference Figure 27 The sample handwritten text trajectory is set with corresponding sample context labels and sample context labels. First, the sample handwritten text trajectory is sampled in time sequence to obtain sample temporal sampling coordinates. Then, the temporal spatial feature extraction model is used to extract features from the sample temporal sampling coordinates to obtain temporal spatial features. The temporal spatial features and sample context labels are input into the fusion prediction model to obtain the predicted sample context. Based on the difference between the predicted sample context and the sample context label, a loss function is calculated, and the temporal spatial feature extraction model and the fusion prediction model are jointly trained using the loss function.

[0487] The embodiment of steps 2610 to 2650 described above jointly trains a temporal-spatial feature extraction model and a fusion prediction model based on a set of sample handwritten text trajectories. Specifically, the sample handwritten text trajectories in the set of sample handwritten text trajectories have sample previous context labels and sample next context labels. The sample handwritten text trajectories are sampled in a temporal sequence to obtain sample temporal coordinates. Feature extraction is performed on the sample temporal coordinates to obtain sample temporal-spatial features of the sample handwritten text trajectories. The sample temporal-spatial features and sample previous context labels are then used as inputs to a fusion prediction model, and the output of the fusion prediction model is the predicted sample next context. Based on the predicted sample next context and the sample next context labels, a loss function is determined, and the parameters of the temporal-spatial feature extraction model and the fusion prediction model are adjusted using the loss function to achieve joint training of the models. The disclosed embodiment jointly trains the temporal-spatial feature extraction model and the fusion prediction model to improve their performance. Ultimately, the predicted next context information corresponding to the target handwritten text trajectory obtained by the model is more accurate, thereby improving the efficiency of the handwritten text prediction method.

[0488] The above is a general description of steps 2610 to 2650. The specific implementation process of steps 2610 to 2650 will be described in detail below.

[0489] In one embodiment, the sample handwritten text trajectory can be determined by phrase segmentation. Specifically, text recognition is performed on all handwritten text trajectories in the sample handwritten text trajectory set to obtain a sample text recognition result. Afterwards, the sample text recognition result is input into the phrase segmentation model to obtain a segmented sample phrase. Multiple time points are selected, and the handwritten text trajectory from the starting time point of the most recent sample phrase before the selected time point to the time point before the selected time point is used as the sample handwritten text trajectory. The handwritten text trajectory before the sample handwritten text trajectory is used as the sample previous context label, and the handwritten text trajectory after the sample handwritten text trajectory is used as the sample following context label.

[0490] In one embodiment, the sample handwritten text trajectory can be determined by breakpoint recognition. Specifically, breakpoint recognition is performed on all handwritten text trajectories in the sample handwritten text trajectory set to obtain recognized sample breakpoints. Multiple time points are selected, and the first sample breakpoint is determined from the recognized sample breakpoints. The first sample breakpoint is a breakpoint whose disconnection time exceeds the first duration, is located before the selected time point, and is the breakpoint closest to the selected time point. The handwritten text trajectory starting from the first sample breakpoint and ending at the selected time point is used as the sample handwritten text trajectory. The handwritten text trajectory before the sample handwritten text trajectory is used as the sample previous context label, and the handwritten text trajectory after the sample handwritten text trajectory is used as the sample next context label.

[0491] In one embodiment, referring to Figure 28 After step 2610, the joint training method of the temporal spatial feature extraction model and the integrated prediction model further includes the following steps:

[0492] Step 2810: Select adjacent stroke pairs to be exchanged from the sample handwritten text trajectories in the sample handwritten text trajectory set;

[0493] Step 2820: swap the adjacent strokes in the adjacent stroke pairs to obtain a handwritten text trajectory after swapping.

[0494] Step 2830: Expand the swapped handwritten text trajectory to the sample handwritten text trajectory set. The swapped handwritten text trajectory has the same sample previous-context label and sample next-context label as the sample handwritten text trajectory.

[0495] Steps 2810 to 2830 are described in detail below.

[0496] In step 2810, select adjacent stroke pairs to be swapped in the sample handwritten text trajectories in the sample handwritten text trajectory set. The adjacent stroke pairs to be swapped refer to the stroke pairs that need to be swapped, and the adjacent stroke pairs to be swapped are located in the same character. In addition, multiple adjacent stroke pairs to be swapped can be set for the same character.

[0497] In step 2820, swap the adjacent strokes to be swapped in the adjacent stroke pairs to be swapped, and obtain the handwritten text trajectory after swapping. The handwritten text trajectory after swapping is the result of swapping the adjacent strokes corresponding to the adjacent stroke pairs to be swapped in the sample handwritten text trajectory.

[0498] It should be noted that the stroke swapping is not only to swap the adjacent strokes to be swapped in the adjacent stroke pairs to be swapped, but also to swap the time points corresponding to the adjacent strokes to be swapped, so as to facilitate the subsequent correspondence between the sampling points and the time points.

[0499] In step 2830, expand the handwritten text trajectory after swapping into the sample handwritten text trajectory set. The handwritten text trajectory after swapping has the same sample above-label and sample below-label as the sample handwritten text trajectory. When expanding the handwritten text trajectory after swapping into the sample handwritten text trajectory set, if the strokes of the handwritten text trajectory are correct, then the strokes of the handwritten text trajectory after swapping are incorrect. Using the handwritten text trajectory with incorrect strokes as the sample handwritten text trajectory to jointly train the temporal-spatial feature extraction model and the fusion prediction model, the trained temporal-spatial feature extraction model and fusion prediction model can predict the target handwritten text trajectory with incorrect strokes, thus expanding the applicable range of handwritten text prediction.

[0500] For Figure 29 the sample handwritten text trajectory set shown, take "生" as the sample handwritten text trajectory. Its corresponding sample above-label is "你", and the sample below-label is "日". Select the fourth stroke "丨" and the fifth stroke "一" in the sample handwritten text trajectory as the adjacent stroke pairs to be swapped. Swap the adjacent strokes to be swapped in the adjacent stroke pairs to be swapped, obtain the handwritten text trajectory after swapping, and expand the handwritten text trajectory after swapping into the sample handwritten text trajectory set. The sample above-label corresponding to the handwritten text trajectory after swapping is "你", and the sample below-label is "日".

[0501] In the embodiment described above, steps 2810 through 2830 swap adjacent strokes corresponding to pairs of adjacent strokes to be swapped in the sample handwritten text trajectory, generating a swapped handwritten text trajectory, and adding the swapped handwritten text trajectory to the set of sample handwritten text trajectories. This stroke swapping process enables the trained temporal-spatial feature extraction model and fusion prediction model to predict target handwritten text trajectories containing incorrect strokes, thereby expanding the scope of application of handwritten text prediction.

[0502] In one embodiment, the disclosed embodiment can select any two strokes in the sample handwritten text trajectory as a stroke pair to be interchanged, and the stroke pair to be interchanged is located in the same character. Afterwards, the strokes to be interchanged in the stroke pair to be interchanged are interchanged, and the obtained interchange result is expanded to the sample handwritten text trajectory set.

[0503] In one embodiment, referring to Figure 30 After step 2610, the joint training method of the temporal spatial feature extraction model and the integrated prediction model further includes the following steps:

[0504] Step 3010: rotating the sample handwritten text trajectories in the sample handwritten text trajectory set by multiple rotation angles to obtain multiple rotated handwritten text trajectories, wherein the multiple rotation angles are in an arithmetic progression;

[0505] Step 3020: Expand the multiple rotated handwritten text trajectories into the sample handwritten text trajectory set, wherein the rotated handwritten text trajectories have the same sample previous-context labels and sample next-context labels as the sample handwritten text trajectories.

[0506] The following is a message description of steps 3010 to 3020.

[0507] In step 3010, the sample handwritten text trajectories in the sample handwritten text trajectory set are rotated at multiple rotation angles to obtain multiple rotated handwritten text trajectories, where the multiple rotation angles are in an arithmetic progression. The rotated handwritten text trajectories are the results obtained after the sample handwritten text trajectories are rotated.

[0508] The rotation of the sample handwritten text trajectory includes both clockwise rotation and counterclockwise rotation. To ensure that the handwritten text prediction method can be applied to handwritten text prediction methods with various angles of inclination, the smaller the tolerance corresponding to multiple rotation angles, the better. However, to improve the model training speed and reduce the processing overhead, the larger the tolerance corresponding to the rotation angle, the better. Therefore, the embodiments of the present disclosure need to select an appropriate tolerance according to the writing habit of the object. In addition, the value range of the rotation angle can also be set according to the writing habit of the object. For example, the object can usually write a text trajectory rotated by 180 degrees only when the touch screen is inverted. In this case, it can be set that the handwritten text prediction application rotates with the terminal direction, and then this angle can be outside the value range of the rotation angle.

[0509] In step 3020, multiple rotated handwritten text trajectories are expanded into the sample handwritten text trajectory set. The rotated handwritten text trajectories have the same sample above-text label and sample below-text label as the sample handwritten text trajectory. By expanding the rotated handwritten text trajectories into the sample handwritten text trajectory set and jointly training the temporal-spatial feature extraction model and the fusion prediction model based on the rotated handwritten text trajectories, the trained temporal-spatial feature extraction model and fusion prediction model can predict the inclined target handwritten text trajectory, thus expanding the applicable range of handwritten text prediction.

[0510] For Figure 31 the sample handwritten text trajectory set shown, use "sheng" as the sample handwritten text trajectory. Its corresponding sample above-text label is "ni", and the sample below-text label is "ri". Rotate the sample handwritten text trajectory at multiple rotation angles to obtain the rotated handwritten text trajectories, and the multiple rotation angles form an arithmetic sequence. Then expand the rotated handwritten text trajectories into the sample handwritten text trajectory set. The corresponding sample above-text label of the rotated handwritten text trajectory is "ni", and the sample below-text label is "ri".

[0511] The embodiments of the above steps 2810 to 2830 rotate the sample handwritten text trajectory based on multiple rotation angles to obtain the rotated handwritten text trajectories, and expand the rotated handwritten text trajectories into the sample handwritten text trajectory set. The angle rotation processing method enables the trained temporal-spatial feature extraction model and fusion prediction model to predict the inclined target handwritten text trajectory, thus expanding the applicable range of handwritten text prediction.

[0512] In step 2620, sample the sample handwritten text trajectory in chronological order to obtain the sample temporal sampling coordinates.

[0513] In one embodiment, a subject can write by moving a finger on the touch screen, thereby obtaining a sample handwritten text trajectory. Alternatively, a subject can write on the touch screen using a stylus, capacitive pen, or other handwriting tool, thereby generating a handwritten text trajectory. The movement speed of the finger, stylus, capacitive pen, or other tool on the touch screen is obtained, and a sampling frequency is set based on the movement speed. Finally, based on the sampling frequency, the sample handwritten text trajectory is sampled to obtain sample time-series sampling coordinates.

[0514] It should be noted that the embodiment of the present disclosure can sample the sample handwritten text trajectory in time sequence by random sampling. The random sampling method does not need to obtain the movement speed of the finger on the touch screen, which can improve the sampling speed of the sample handwritten text trajectory.

[0515] In one embodiment, the sampling frequency corresponding to sampling the sample handwritten text trajectory can be set to a fixed value. Specifically, the sampling frequency can be determined based on the average movement speed of all objects on the touch screen in the historical database, so that the sampling frequency can be suitable for most people, while improving the sampling speed of the sample handwritten text trajectory.

[0516] In one embodiment, referring to Figure 32 , step 2620 includes:

[0517] Step 3210: Sample the sample handwritten text trajectory in time sequence according to multiple sampling frequencies to obtain multiple groups of sample time series sampling coordinates.

[0518] In step 2640, the sample temporal spatial features and the sample context labels are input into the fusion prediction model to obtain the predicted sample context.

[0519] Corresponding to step 3210, step 2640 includes:

[0520] Step 3220: For each set of sample time series sampling coordinates, the sample time series spatial features and sample context labels corresponding to the set of sample time series sampling coordinates are input into the fusion prediction model to obtain the sample context predicted for the set of sample time series sampling coordinates.

[0521] In step 3650, based on the difference between the predicted sample context and the sample context label, a loss function is calculated, and the temporal spatial feature extraction model and the fusion prediction model are jointly trained using the loss function.

[0522] Corresponding to step 3210 and step 3220, step 2650 includes:

[0523] Step 3230: Calculate the loss function corresponding to the set of sample time series sampling coordinates based on the difference between the sample context predicted for the set of sample time series sampling coordinates and the sample context label.

[0524] Steps 3210 to 3230 are described in detail below.

[0525] In step 3210, the sample handwritten text trajectory is temporally sampled at various sampling frequencies to obtain multiple sets of sample temporal sampling coordinates. The groups of sample temporal sampling coordinates correspond to the sampling frequencies, and each set of sample temporal sampling coordinates is the result of temporally sampling the sample handwritten text trajectory at the corresponding sampling frequency.

[0526] Different objects have different font sizes when writing on a touch screen. Therefore, the disclosed embodiment sets multiple sampling frequencies and samples the sample handwritten text trajectory in time sequence based on the sampling frequencies to achieve the same effect when sampling fonts of different sizes.

[0527] In step 3220, for each set of sample time series sampling coordinates, the corresponding sample time series spatial features and sample context labels are input into the fusion prediction model to obtain the sample context predicted for the set of sample time series sampling coordinates. For each set of sample time series sampling coordinates, the corresponding sample time series spatial features and sample context labels are input into the fusion prediction model to obtain the predicted sample context. The predicted sample context corresponds one-to-one to the set of sample time series sampling coordinates.

[0528] It should be noted that the sample temporal spatial features corresponding to the multiple groups of sample temporal sampling coordinates obtained through the same sample writing text trajectory are different, while the sample previous label and the sample next label remain consistent.

[0529] In step 3230, a loss function corresponding to the set of sample time series sampling coordinates is calculated based on the difference between the sample context predicted for the set of sample time series sampling coordinates and the sample context label. The loss function corresponds one-to-one to each set of sample time series sampling coordinates, with each set of time series sampling coordinates corresponding to a loss function.

[0530] During the joint training process of the time series spatial feature extraction model and the fusion prediction model, their parameters need to be adjusted and optimized in real time based on the loss function. Therefore, the time series spatial feature extraction model and the fusion prediction model used to process the time series sampling coordinates of each set of samples may be different.

[0531] Reference Figure 33In the embodiment of the present disclosure, multiple sampling frequencies are provided, namely sampling frequency A, sampling frequency B, ..., sampling frequency N. Based on the multiple frequencies, the sample handwritten text trajectory is sampled in time sequence, and the sample time series sampling coordinate A corresponding to sampling frequency A, the sample time series sampling coordinate B corresponding to sampling frequency B, ..., and the sample time series sampling coordinate N corresponding to sampling frequency N are obtained. Feature extraction is then performed on multiple groups of sample time series sampling coordinates to obtain multiple sample time series sampling point features corresponding to the sample time series sampling coordinates. The sample time series sampling point features are then input into the time series space feature extraction model to obtain multiple time series space features corresponding to the sample time series sampling coordinates. Then, for each group of sample time series sampling coordinates, the sample time series space features corresponding to the group of sample time series sampling coordinates and the sample context labels are input into the fusion prediction model to obtain the sample context predicted for the group of sample time series sampling coordinates. Based on the difference between the multiple predicted sample contexts and their corresponding sample context labels, a loss function is calculated, and the time series space feature extraction model and the fusion prediction model are jointly trained based on the loss function.

[0532] In the embodiment of steps 3210 to 3230 described above, sample handwritten text trajectories are temporally sampled at multiple sampling frequencies to obtain multiple sets of sample time-series coordinates. The temporal-spatial feature extraction model and fusion prediction model are then trained using these multiple sets of sample time-series coordinates corresponding to the different sampling frequencies. The trained temporal-spatial feature extraction model and fusion prediction model are ultimately capable of predicting target handwritten text trajectories in any font size, thereby expanding the applicability of handwritten text prediction.

[0533] In step 2630, the sample temporal spatial features of the sample handwritten text trajectory are extracted from the sample temporal sampling coordinates.

[0534] In one embodiment, the sample time series sampling coordinates are the coordinates of multiple sample sampling points in time sequence, referring to Figure 34 , step 2630 includes:

[0535] Step 3410: Obtain sample sampling point features of the sample handwritten text trajectory based on the sample time series sampling coordinates;

[0536] Step 3420: Input the sample sampling point features into the time series space feature extraction model to obtain the sample time series space features.

[0537] Step 3410 and step 3420 are described in detail below.

[0538] In step 3410, sample sampling point features of the sample handwritten text trajectory are obtained based on the sample time-series sampling coordinates. Sample sampling point features refer to the features of each sampling point corresponding to the sample handwritten text trajectory, and the sample sampling point features include at least positional features and time-series features of the sampling points. In addition, sample sampling point features may also include stroke features, character features, and word features. For example, sample sampling point features may indicate the start and end of a stroke, or the start and end of a character or word.

[0539] In step 3420, the sample sampling point features are input into the temporal-spatial feature extraction model to obtain sample temporal-spatial features. The temporal-spatial feature model processes the sample sampling point features, converting them into sample temporal-spatial features to meet the processing requirements of the fusion prediction model. The sample temporal-spatial features represent the temporal and spatial features of the sample handwritten text trajectory. They can be obtained by concatenating the temporal and spatial features or by fusing them.

[0540] The temporal spatial feature extraction model can be an encoder model, a mapping model, a neural network model, etc. The temporal spatial feature extraction model requires the same temporal spatial feature extraction model architecture as that in step 1120 but different parameters.

[0541] Reference Figure 27 , feature extraction is performed on the sample time series sampling coordinates to determine the sample sampling point features of the sample handwritten text trajectory. Then, the sample sampling point features are input into the time series space feature extraction model to obtain the sample time series space features.

[0542] Steps 3410 and 3420 above first process the sample temporal sampling coordinates to obtain sample sampling point features of the sample handwritten text trajectory. These sample sampling point features are then input into the temporal-spatial feature extraction model to obtain sample temporal-spatial features. Extracting the sample sampling point features and setting up the temporal-spatial feature extraction model allows for more precise processing of the sample temporal sampling coordinates. Consequently, the sample temporal-spatial features can more accurately represent the temporal and spatial features of the sample handwritten text trajectory.

[0543] The above is a general description of step 3410 and step 3420. The specific implementation process of step 3410 and step 3420 is described in detail below.

[0544] In step 3410, based on the sample time series sampling coordinates, the sample sampling point features of the sample handwritten text trajectory are obtained.

[0545] In one embodiment, referring to Figure 35 , step 3410 includes:

[0546] Step 3510: Determine a sample normalized displacement difference of the sample sampling point based on the third coordinate of the sample sampling point and the fourth coordinate of the sample sampling point preceding the sample sampling point;

[0547] Step 3520: Obtain the sample pen lift and drop mark of the sample sampling point;

[0548] Step 3530: Obtain the first sampling point identifier of the sample stroke, where the first sampling point identifier of the sample stroke indicates whether the sample sampling point is the first sample sampling point in the stroke where the sample sampling point is located;

[0549] Step 3540: Generate sample sampling point features of the sample handwritten text trajectory based on the sample normalized displacement difference, the sample pen lift and drop identifiers, and the sample stroke first sampling point identifier.

[0550] Steps 3510 to 3540 are described in detail below.

[0551] In step 3510, a sample normalized displacement difference of the sample sampling point is determined based on the third coordinate of the sample sampling point and the fourth coordinate of the sample sampling point immediately preceding the sample sampling point. The third coordinate is the position information of the sample sampling point, and the fourth coordinate is the position information of the sample sampling point immediately preceding the sample sampling point. The position information may include the horizontal and vertical coordinates of the sample sampling point. In addition to rectangular coordinates, the position information may also be expressed in polar coordinates or natural coordinates.

[0552] The previous sample sampling point of a sample sampling point is the previous sample sampling point of the sample sampling point in the time sequence. For example, if the time information corresponding to the current sample sampling point is the t-th moment, then the previous sample sampling point is the sampling point at the t-1th moment.

[0553] In step 3520, a sample pen-up / pen-down flag is obtained for the sample sampling point. The sample pen-up / pen-down flag is used to indicate the pen-up / pen-down state of the sample handwritten text trajectory. The pen-up / pen-down flag has at least two values, one for indicating the pen-up state and the other for indicating the pen-down state of the sample handwritten text trajectory.

[0554] For a stroke, the subject usually needs to lift the finger or handwriting tool off the touch screen at the last point, so the last stroke is the pen-up state, and the remaining points of the stroke require the handwriting or handwriting tool to always maintain contact with the touch screen, so the first point and the middle points of the stroke are the pen-down state.

[0555] In step 3530, the sample stroke first sampling point identifier is obtained, and the sample stroke first sampling point identifier indicates whether the sample sampling point is the first sample sampling point in the stroke where the sample sampling point is located. The sample stroke first sampling point identifier can indicate whether the sample sampling point is the first point in the stroke where the sample sampling point is located.

[0556] The sample stroke first sampling flag is set to at least two values, one of which is used to indicate that the sample sampling point is the first point in the stroke where it is located, and the other is used to indicate that the sample sampling point is not the first point in the stroke where it is located.

[0557] In step 3540, sample sampling point features of the sample handwritten text trajectory are generated based on the sample normalized displacement difference, the sample pen-up and pen-down markers, and the sample stroke first sampling point markers. The sample normalized displacement difference, the sample pen-up and pen-down markers, and the sample stroke first sampling point markers are used as the sample sampling point features of the sample handwritten text trajectory. The sample normalized displacement difference is used to represent the positional features of the corresponding sample sampling point, while the sample pen-up and pen-down markers and the sample stroke first sampling point markers based on multiple sample sampling points can confirm the start and end of the stroke in which they are located.

[0558] It should be noted that the sample sampling point features only need to represent the spatial and temporal features of the sample handwritten text trajectory. Therefore, the sample sampling point features can be determined solely based on normalized displacement differences, or the sample sampling point features can also be the position coordinates of the sample sampling point and its corresponding previous sample sampling point. However, the specific content of the sample sampling point features must be consistent with the sampling point features corresponding to the target handwritten text trajectory.

[0559] Reference Figure 36 The sample normalized displacement difference of the sample sampling point is determined based on the position coordinates of the sample sampling point and its corresponding previous sample sampling point, and then the sample pen lift and drop identifier and the sample stroke first sampling point identifier are obtained, so as to generate the sample sampling point feature based on the sample normalized displacement difference, the sample pen lift and drop identifier, and the sample stroke first sampling point identifier.

[0560] The embodiment of the above steps 3510 to 3540 generates sample sampling point features based on the sample normalized displacement difference, the sample pen lift and drop mark, and the sample stroke first sampling point mark, and trains the temporal feature extraction model and the fusion prediction model based on the sample sampling point features. The normalized displacement difference can represent the positional features of each sampling point corresponding to the target handwritten text trajectory, the pen lift and drop mark and the stroke first sampling point mark are used to represent the features of the stroke where the sampling point is located, and based on the arrangement order of the normalized displacement difference, the pen lift and drop mark, and the stroke first sampling point mark corresponding to each sampling point, the temporal information of the sampling point can be determined. In addition to the spatial information and temporal information of the target handwritten text trajectory, the sampling point features also contain stroke information, which can make the performance of the trained temporal feature extraction model and the fusion prediction model better, thereby improving the prediction accuracy of the target handwritten text trajectory.

[0561] The above is a general description of steps 3510 to 3540. The specific implementation process of steps 3510 to 3530 will be described in detail below.

[0562] In step 3510 , a sample normalized displacement difference of the sample sampling point is determined based on the third coordinate of the sample sampling point and the fourth coordinate of the previous sample sampling point.

[0563] In one embodiment, the third coordinate includes a third horizontal coordinate and a third vertical coordinate, and the fourth coordinate includes a fourth horizontal coordinate and a fourth vertical coordinate. Figure 37 , step 3510 includes:

[0564] Step 3710: Determine a third difference between a third horizontal coordinate of the sample sampling point and a fourth horizontal coordinate of the sample sampling point preceding the sample sampling point, and a fourth difference between a third vertical coordinate of the sample sampling point and a fourth vertical coordinate of the sample sampling point preceding the sample sampling point.

[0565] Step 3720: Obtain the maximum value and minimum value of the sample vertical coordinate of the sample handwritten text trajectory in the sample handwritten text trajectory set;

[0566] Step 3730: Determine the sample vertical coordinate span based on the maximum value and the minimum value of the sample vertical coordinate;

[0567] Step 3740: Determine the sample normalized horizontal coordinate displacement differential of the sample sampling point based on the third difference and the sample vertical coordinate span, and determine the sample normalized vertical coordinate displacement differential of the sampling point based on the fourth difference and the sample vertical coordinate span;

[0568] Step 3750: Determine the sample normalized displacement difference of the sample sampling point based on the sample normalized horizontal coordinate displacement difference and the sample normalized vertical coordinate displacement difference.

[0569] Steps 3710 to 3750 are described in detail below.

[0570] In step 3710, a third difference between the third abscissa of the sample sampling point and the fourth abscissa of the sample sampling point immediately preceding the sample sampling point, and a fourth difference between the third ordinate of the sample sampling point and the fourth ordinate of the sample sampling point immediately preceding the sample sampling point, are determined. The third difference refers to the difference between the abscissas of the sample sampling point and its corresponding immediately preceding sample sampling point, i.e., the difference between the third abscissa and the fourth abscissa. The fourth difference refers to the difference between the ordinates of the sample sampling point and its corresponding immediately preceding sample sampling point, i.e., the difference between the third ordinate and the fourth ordinate.

[0571] In step 3720, the maximum and minimum sample ordinate values ​​of the sample handwritten text trajectories in the sample handwritten text trajectory set are obtained. The maximum sample ordinate value refers to the maximum value of the ordinate of the sampling points corresponding to all the sample handwritten text trajectories in the sample handwritten text trajectory set. The minimum sample ordinate value refers to the minimum value of the ordinate of the sampling points corresponding to all the sample handwritten text trajectories in the sample handwritten text trajectory set.

[0572] Different from the maximum and minimum ordinate values ​​of the target handwritten text trajectory, the maximum and minimum ordinate values ​​of the sample handwritten text trajectory corresponding to the sample handwritten text trajectory are determined based on the sampling points corresponding to all the sample handwritten text trajectories in the sample handwritten text trajectory set.

[0573] In step 3730, the sample vertical coordinate span is determined based on the sample vertical coordinate maximum and sample vertical coordinate minimum. The sample vertical coordinate span refers to the vertical coordinate span of all sample handwritten text tracks in the sample handwritten text track set. The sample vertical coordinate span can be specifically expressed as the difference between the sample vertical coordinate maximum and the sample vertical coordinate minimum.

[0574] In step 3740, the sample normalized horizontal coordinate displacement difference of the sample sampling point is determined based on the third difference and the sample vertical coordinate span, and the sample normalized vertical coordinate displacement difference of the sampling point is determined based on the fourth difference and the sample vertical coordinate span. The sample normalized vertical coordinate displacement difference refers to the result obtained by normalizing the displacement difference between the sample sampling point and the previous sample sampling point in the vertical coordinate axis direction. The sample normalized horizontal coordinate displacement difference refers to the result obtained by normalizing the displacement difference between the sample sampling point and the previous sample sampling point in the horizontal coordinate axis direction. Specifically, the embodiment of the present disclosure normalizes the displacement differences, i.e., the third difference and the fourth difference, by the sample vertical coordinate span. The sample normalized horizontal coordinate displacement difference is the ratio of the third difference to the sample vertical coordinate span, and the sample normalized vertical coordinate displacement difference is the ratio of the fourth difference to the sample vertical coordinate span.

[0575] In step 3750, a sample normalized displacement difference of the sample sampling point is determined based on the sample normalized horizontal displacement difference and the sample normalized vertical displacement difference. The sample normalized displacement difference of the sample sampling point includes its corresponding sample normalized horizontal displacement difference and sample normalized vertical displacement difference.

[0576] If the sample normalized horizontal coordinate displacement difference is ΔX t , the sample normalized vertical coordinate displacement difference is ΔY t , then the normalized displacement difference of the sampling point can be expressed as (ΔX t ,ΔY t ).

[0577] Reference Figure 38 , obtain the third ordinate and third abscissa of the sample sampling point and the fourth ordinate and fourth abscissa of the corresponding previous sample sampling point. Based on the difference between the third abscissa and the fourth abscissa, obtain the third difference, and based on the difference between the third ordinate and the fourth ordinate, obtain the fourth difference. In addition, based on the sampling points corresponding to all sample handwriting text trajectories in the sample handwriting text trajectory set, determine the maximum value of the sample ordinate and the minimum value of the sample ordinate, and use the difference between the maximum value of the sample ordinate and the minimum value of the sample ordinate as the sample ordinate span. Afterwards, normalize the third difference based on the sample ordinate span to obtain the sample normalized abscissa displacement difference, and normalize the fourth difference based on the sample ordinate span to obtain the sample normalized ordinate displacement difference. Finally, determine the sample normalized displacement difference of the sampling point based on the sample normalized abscissa displacement difference and the sample normalized ordinate displacement difference.

[0578] In the above-described embodiment of steps 3710 to 3750, the third and fourth differences of the sample sampling points are normalized based on the sample vertical coordinate span of the sample handwritten text trajectory, resulting in a sample normalized horizontal coordinate displacement differential and a sample normalized vertical coordinate displacement differential, thereby determining the sample normalized displacement differential of the sample sampling point. Compared to directly using the sampling point coordinates or displacement differential as the sample sampling point features, the sample normalized displacement differential obtained after normalization can reduce the occurrence of problems such as overfitting without losing the displacement differential information, thereby improving the performance of the trained time series feature extraction model and the fusion prediction model.

[0579] In one embodiment, a third pressure at a sample sampling point and a fourth pressure at a sample sampling point immediately preceding the sample sampling point are obtained. If the third pressure is less than the fourth pressure, the sample pen-up / down indicator is determined to have a sixth value indicating pen-up. If the third pressure is not less than the fourth pressure, the sample pen-up / down indicator is determined to have a seventh value indicating pen-down. The sixth value of the sample pen-up / down indicator is typically equal to the first value of the pen-up / down indicator, and the seventh value of the sample pen-up / down indicator is typically equal to the second value of the pen-up / down indicator.

[0580] In one embodiment, before determining the sample normalized displacement difference, a sample interruption sampling point is set for the position where the stroke of the sample handwritten text trajectory is interrupted, wherein the horizontal coordinate of the sample interruption sampling point is outside the horizontal coordinate value range of the sample handwritten text trajectory, and the vertical coordinate of the sample sampling point is outside the vertical coordinate value range of the sample handwritten text trajectory. If the sample sampling point immediately preceding the sample sampling point is a sample interruption sampling point, the sample stroke first sampling point identifier is set to an eighth value. If the sample sampling point immediately preceding the sample sampling point is not a sample interruption sampling point, the sample stroke first sampling point identifier is set to a ninth value. The eighth value of the sample stroke first sampling point identifier is generally equal to the third value of the stroke first sampling point identifier, and the ninth value of the sample stroke first sampling point identifier is generally equal to the fourth value of the stroke first sampling point identifier.

[0581] In one embodiment, the specific architecture of the time series feature extraction model and the fusion prediction model is as follows: Figure 4 In the embodiment of the present disclosure, the sample sampling point features are sequentially input into the long short-term memory network, the first linear layer, the convolution layer, and the maximum pooling layer of the temporal feature extraction model to obtain the sample temporal spatial features.

[0582] To predict the context of a sample, the sample embedding vector corresponding to the sample's preceding label is first determined. The sample position encoding information for each character of the preceding label is then added to the sample embedding vector. The sample embedding vector and the sample's temporal-spatial features are then processed using a fusion prediction model. Specifically, the sample embedding vector is input into a multi-head attention model to obtain a sample attention vector. The sample attention vector is then superimposed with the sample embedding vector to obtain a first sample superposition vector. The first sample superposition vector is normalized to obtain a first sample normalized vector. The first sample normalized vector and the sample temporal-spatial features are input into a fusion layer to obtain a sample fusion vector. The first sample normalized vector and the sample temporal-spatial features are then input into a fusion layer to obtain a fourth sample intermediate vector. The fourth sample intermediate vector, the first sample normalized vector, and the sample temporal-spatial features are then superimposed to obtain a second sample superposition vector. The second sample superposition vector is then normalized to obtain a second sample normalized vector. The second sample normalized vector is input into a second linear layer to obtain a fifth sample intermediate vector. The second sample normalized vector and the fifth sample intermediate vector are superimposed to obtain a third sample superposition vector. Then the third superposition vector is normalized to obtain the fusion vector. Finally, the sample fusion vector is input into the third linear layer and the decision layer in sequence to obtain the predicted sample context.

[0583] Implementation details of the handwritten text prediction method of the disclosed embodiment

[0584] Refer to the following Figure 39 , which illustrates in detail the implementation details of the handwritten text prediction method of the embodiment of the present disclosure.

[0585] In step 3910 , a sample handwritten text trajectory set is obtained, wherein the sample handwritten text trajectory in the sample handwritten text trajectory set has a sample previous context label and a sample next context label.

[0586] In step 3920, the sample handwritten text trajectory is subjected to stroke swapping, multi-angle rotation, and multi-sampling frequency to obtain a new sample handwritten text trajectory, and the new sample handwritten text trajectory is added to the sample handwritten text trajectory set.

[0587] In one embodiment, adjacent stroke pairs to be exchanged are selected from sample handwritten text trajectories in the sample handwritten text trajectory set, and the adjacent strokes to be exchanged in the adjacent stroke pairs to be exchanged are exchanged to obtain a handwritten text trajectory after exchange.

[0588] In one embodiment, a sample handwritten text trajectory in a sample handwritten text trajectory set is rotated at multiple rotation angles to obtain multiple rotated handwritten text trajectories, where the multiple rotation angles are in an arithmetic progression. The multiple rotated handwritten text trajectories are added to the sample handwritten text trajectory set, and the rotated handwritten text trajectories have the same sample context labels and sample context labels as the sample handwritten text trajectories.

[0589] In step 3931, the sample handwritten text trajectory is sampled in time sequence to obtain the sample time series sampling coordinates.

[0590] In one embodiment, the sample handwritten text trajectories are sampled in time sequence according to multiple sampling frequencies to obtain multiple sets of sample time sequence sampling coordinates.

[0591] In step 3932, the sample normalized displacement difference of the sample sampling point is determined based on the third coordinate of the sample sampling point and the fourth coordinate of the sample sampling point immediately preceding the sample sampling point. A sample pen-up / pen-down marker is obtained for the sample sampling point. The sample stroke first sampling point marker is obtained, indicating whether the sample sampling point is the first sample sampling point in the stroke in which the sample sampling point is located. Based on the sample normalized displacement difference, the sample pen-up / pen-down marker, and the sample stroke first sampling point marker, sample sampling point features of the sample handwritten text trajectory are generated.

[0592] In one embodiment, a third difference between the third horizontal coordinate of the sample sampling point and the fourth horizontal coordinate of the sample sampling point preceding the sample sampling point, as well as a fourth difference between the third vertical coordinate of the sample sampling point and the fourth vertical coordinate of the sample sampling point preceding the sample sampling point, are determined. The maximum and minimum vertical coordinate values ​​of the sample handwritten text trajectories in the sample handwritten text trajectory set are then obtained, and the vertical coordinate span of the sample is determined based on the maximum and minimum vertical coordinate values. Based on the third difference and the vertical coordinate span, the sample normalized horizontal coordinate displacement differential of the sample sampling point is determined, and based on the fourth difference and the vertical coordinate span, the sample normalized vertical coordinate displacement differential of the sample sampling point is determined. Finally, based on the sample normalized horizontal coordinate displacement differential and the sample normalized vertical coordinate displacement differential, the sample normalized displacement differential of the sample sampling point is determined.

[0593] In step 3933, based on the sample time series sampling coordinates, the sample sampling point features of the sample handwritten text trajectory are obtained, and the sample sampling point features are input into the time series space feature extraction model.

[0594] In step 3934, the sample sampling point features are input into the time series space feature extraction model to obtain the sample time series space features.

[0595] In step 3941, a sample context label corresponding to the sample handwritten text trajectory is obtained.

[0596] In step 3942, the sample temporal spatial features and the sample context labels are input into the fusion prediction model to obtain the predicted sample context.

[0597] In step 3943 and step 3944, the loss function is calculated based on the difference between the predicted sample context and the sample context label, and the temporal spatial feature extraction model and the fusion prediction model are jointly trained through the loss function.

[0598] In step 3951, the input handwritten text trajectory is obtained.

[0599] In steps 3952 and 3953, the input handwritten text trajectory is subjected to semantic segmentation or breakpoint recognition to obtain the target handwritten text trajectory.

[0600] In one embodiment, text recognition is performed on the handwritten text trajectory input after the start of handwritten text input to obtain an input text recognition result. The input text recognition result is input into a phrase segmentation model to obtain segmented phrases. The input handwritten text trajectory starting from the start time point of the most recent phrase before the current time point is used as the target handwritten text trajectory.

[0601] In one embodiment, breakpoint identification is performed on the handwritten text trajectory input after handwritten text input begins to obtain identified breakpoints. A first breakpoint is determined from the identified breakpoints. The first breakpoint is the breakpoint closest to the current time point among the breakpoints whose break time exceeds the first duration. The handwritten text trajectory input starting from the first breakpoint is used as the target handwritten text trajectory.

[0602] In step 3961, the target handwritten text trajectory is sampled in time sequence to obtain time series sampling coordinates.

[0603] In step 3962, the temporal spatial features of the target handwritten text trajectory are extracted from the temporal sampling coordinates. Specifically, the sampling point features of the target handwritten text trajectory are obtained based on the temporal sampling coordinates.

[0604] In step 3971, the context information of the target handwritten text trajectory is obtained.

[0605] In step 3972, an embedding vector corresponding to the above information is determined, and the position encoding information of each character in the above information is added to the embedding vector.

[0606] In step 3863 and step 3864, the sampling point features are input into the time series space feature extraction model to obtain the time series space features.

[0607] Specifically, the sampling point features are input into the long short-term memory network to obtain the first intermediate vector, which is then input into the convolutional layer to obtain the temporal spatial features.

[0608] In step 3981 and step 3952, the embedding vector and temporal spatial features are input into the fusion prediction model to obtain the predicted text information.

[0609] In one embodiment, the embedding vector is input into a multi-head attention model to obtain an attention vector. The attention vector and temporal spatial features are input into a fusion layer to obtain a fused vector. The fused vector is input into a decision layer to obtain predicted text information.

[0610] Description of the apparatus and device of the present disclosure

[0611] It is to be understood that, although the steps in the above-mentioned flowcharts are shown in sequence according to the arrow representations, these steps are not necessarily performed in sequence according to the order represented by the arrows. Unless otherwise specified in the present embodiment, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above-mentioned flowcharts may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0612] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the characteristics of the task content, such as task content attribute information or attribute information sets, the permission or consent of the task content will be obtained first, and the collection, use and processing of such data will comply with relevant laws, regulations and standards. In addition, when the embodiment of the present application needs to obtain task content attribute information, it will obtain a separate permission or separate consent for the task content through a pop-up window or jump to a confirmation page. After clearly obtaining a separate permission or separate consent for the task content, the necessary task content-related data for the normal operation of the embodiment of the present application will be obtained.

[0613] Figure 40 This is a schematic diagram of the structure of a handwritten text prediction device 4000 provided in an embodiment of the present disclosure. The handwritten text prediction device 4000 includes:

[0614] A first acquisition unit 4010 is used to acquire a target handwritten text trajectory;

[0615] A first sampling unit 4020 is configured to perform temporal sampling on the target handwritten text trajectory to obtain temporal sampling coordinates;

[0616] A first extraction unit 4030 is configured to extract temporal spatial features of the target handwritten text trajectory from the temporal sampling coordinates;

[0617] The second acquiring unit 4040 is configured to acquire context information of the target handwritten text track from a recognition result of a handwritten text input before the target handwritten text track;

[0618] The first input unit 4050 is used to input the temporal spatial features and the above context information into a fusion prediction model to obtain predicted text information.

[0619] Optionally, the first acquiring unit 4010 is specifically configured to:

[0620] Performing text recognition on the input handwritten text trajectory after the handwritten text input starts, and obtaining the input text recognition result;

[0621] Input the input text recognition result into the phrase segmentation model to obtain the segmented phrases;

[0622] The input handwritten text trajectory starting from the starting time point of the most recent phrase before the current time point is used as the target handwritten text trajectory.

[0623] Optionally, the first acquiring unit 4010 is further configured to:

[0624] Performing breakpoint recognition on the handwritten text track that has been input after the handwritten text input starts, and obtaining the recognized breakpoints;

[0625] Determine a first breakpoint from the identified breakpoints, where the first breakpoint is the breakpoint closest to the current time point among the breakpoints whose disconnection time exceeds the first duration;

[0626] The input handwritten text trajectory starting from the first breakpoint is used as the target handwritten text trajectory.

[0627] Optionally, the target handwritten text trajectory is formed by movement of a finger on the touch screen;

[0628] The first sampling unit 4020 is specifically configured to:

[0629] Get the movement speed of the finger on the touch screen;

[0630] Determine the sampling frequency based on the moving speed;

[0631] Based on the sampling frequency, the target handwritten text trajectory is sampled to obtain the time series sampling coordinates.

[0632] Optionally, the time series sampling coordinates are coordinates of a plurality of sampling points in time sequence;

[0633] The first extraction unit 4030 is specifically configured to:

[0634] Based on the time series sampling coordinates, the sampling point features of the target handwritten text trajectory are obtained;

[0635] The sampling point features are input into the time series space feature extraction model to obtain the time series space features.

[0636] Optionally, the handwritten text prediction device 4000 further includes:

[0637] A third acquiring unit is configured to acquire a set of sample handwritten text trajectories, wherein the sample handwritten text trajectories in the set of sample handwritten text trajectories have a sample previous context label and a sample next context label;

[0638] The second sampling unit is used to sample the sample handwritten text trajectory in time sequence to obtain the sample time series sampling coordinates;

[0639] A second extraction unit is used to extract the sample time sequence spatial features of the sample handwritten text trajectory from the sample time sequence sampling coordinates;

[0640] The second input unit is used to input the sample temporal spatial features and the sample context label into the fusion prediction model to obtain the predicted sample context;

[0641] The loss calculation unit is used to calculate the loss function based on the difference between the predicted sample context and the sample context label, and jointly train the temporal spatial feature extraction model and the fusion prediction model through the loss function.

[0642] Optionally, the handwritten text prediction device 4000 further includes:

[0643] A selection unit, configured to select adjacent stroke pairs to be exchanged from sample handwritten text trajectories in the sample handwritten text trajectory set;

[0644] An interchanging unit, configured to interchange adjacent strokes in a pair of adjacent strokes to be interchanged, to obtain a handwritten text trajectory after interchanging;

[0645] The first expansion unit is used to expand the swapped handwritten text trajectory into the sample handwritten text trajectory set, where the swapped handwritten text trajectory has the same sample previous-context label and sample next-context label as the sample handwritten text trajectory.

[0646] Optionally, the handwritten text prediction device 4000 further includes:

[0647] a rotation unit, configured to rotate the sample handwritten text trajectories in the sample handwritten text trajectory set by multiple rotation angles to obtain multiple rotated handwritten text trajectories, wherein the multiple rotation angles are in an arithmetic progression;

[0648] The second expansion unit is configured to expand the plurality of rotated handwritten text trajectories into the sample handwritten text trajectory set, wherein the rotated handwritten text trajectories have the same sample previous-context labels and sample next-context labels as the sample handwritten text trajectories.

[0649] Optionally, the second sampling unit is specifically configured to: perform time-sequential sampling on the sample handwritten text trajectory according to a plurality of sampling frequencies to obtain a plurality of groups of sample time-series sampling coordinates;

[0650] The second input unit is specifically configured to: for each set of sample time series sampling coordinates, input the sample time series spatial features and sample context labels corresponding to the set of sample time series sampling coordinates into the fusion prediction model to obtain the sample context predicted for the set of sample time series sampling coordinates;

[0651] The loss calculation unit is specifically used to calculate the loss function corresponding to the set of sample time series sampling coordinates based on the difference between the sample context predicted for the set of sample time series sampling coordinates and the sample context label.

[0652] Optionally, the sample time series sampling coordinates are coordinates of a plurality of sample sampling points in time sequence;

[0653] The second extraction unit is specifically used for:

[0654] Based on the sample time series sampling coordinates, the sample sampling point features of the sample handwritten text trajectory are obtained;

[0655] The sample sampling point features are input into the time series space feature extraction model to obtain the sample time series space features.

[0656] Optionally, the second extraction unit is further configured to:

[0657] Determine a sample normalized displacement difference of the sample sampling point based on a third coordinate of the sample sampling point and a fourth coordinate of a sample sampling point preceding the sample sampling point;

[0658] Obtain the sample pick-up and drop pen identification of the sample sampling point;

[0659] Obtaining a first sampling point identifier of a sample stroke, where the first sampling point identifier of the sample stroke indicates whether the sample sampling point is the first sample sampling point in the stroke where the sample sampling point is located;

[0660] Based on the sample normalized displacement difference, the sample pen lift and drop identifier, and the sample stroke first sampling point identifier, the sample sampling point features of the sample handwritten text trajectory are generated.

[0661] Optionally, the third coordinate includes a third abscissa and a third ordinate, and the fourth coordinate includes a fourth abscissa and a fourth ordinate;

[0662] The second extraction unit is further configured to:

[0663] Determine a third difference between a third abscissa of the sample sampling point and a fourth abscissa of a previous sample sampling point, and a fourth difference between a third ordinate of the sample sampling point and a fourth ordinate of the previous sample sampling point;

[0664] Obtaining a maximum value of a sample vertical coordinate and a minimum value of a sample vertical coordinate of a sample handwritten text trajectory in a sample handwritten text trajectory set;

[0665] Determine the sample ordinate span based on the sample ordinate maximum value and the sample ordinate minimum value;

[0666] Determining a sample normalized abscissa displacement differential of the sample sampling point based on the third difference and the sample ordinate span, and determining a sample normalized ordinate displacement differential of the sampling point based on the fourth difference and the sample ordinate span;

[0667] Based on the sample normalized horizontal coordinate displacement difference and the sample normalized vertical coordinate displacement difference, the sample normalized displacement difference of the sample sampling point is determined.

[0668] Optionally, the first extraction unit 4030 is further configured to:

[0669] Determine a normalized displacement difference of the sampling point based on a first coordinate of the sampling point and a second coordinate of a sampling point preceding the sampling point;

[0670] Get the pen lift and drop mark of the sampling point;

[0671] Obtaining a stroke first sampling point identifier, where the stroke first sampling point identifier indicates whether the sampling point is the first sampling point in the stroke where the sampling point is located;

[0672] The sampling point features of the target handwritten text trajectory are generated based on the normalized displacement difference, pen lift and drop marks, and stroke first sampling point marks.

[0673] Optionally, the first coordinate includes a first abscissa and a second ordinate, and the second coordinate includes a second abscissa and a second ordinate;

[0674] The first extraction unit 4030 is further specifically configured to:

[0675] Determine a first difference between a first abscissa of the sampling point and a second abscissa of a sampling point preceding the sampling point, and a second difference between a first ordinate of the sampling point and a second ordinate of the sampling point preceding the sampling point;

[0676] Obtain the maximum and minimum ordinate values ​​of the target handwritten text trajectory;

[0677] Determine the vertical coordinate span based on the vertical coordinate maximum and vertical coordinate minimum;

[0678] Determining a normalized abscissa displacement differential of the sampling point based on the first difference and the ordinate span, and determining a normalized ordinate displacement differential of the sampling point based on the second difference and the ordinate span;

[0679] Based on the normalized horizontal coordinate displacement difference and the normalized vertical coordinate displacement difference, the normalized displacement difference of the sampling point is determined.

[0680] Optionally, the first extraction unit 4030 is further configured to:

[0681] Obtaining a first pressure at a sampling point and a second pressure at a sampling point preceding the sampling point;

[0682] If the first pressure is less than the second pressure, the pen-lifting / lowering flag is determined to be a first value indicating pen-lifting;

[0683] If the first pressure is not less than the second pressure, the pen-up / pen-down flag is determined as a second value indicating pen-down.

[0684] Optionally, the handwritten text prediction device 4000 further includes:

[0685] an interruption setting unit, configured to set an interruption sampling point at a position where a stroke of a target handwritten text trajectory is interrupted, wherein the abscissa of the interruption sampling point is outside a value range of the abscissa of the target handwritten text trajectory, and the ordinate of the interruption sampling point is outside a value range of the ordinate of the target handwritten text trajectory;

[0686] An adding unit, used to add an interrupted sampling point to multiple sampling points;

[0687] The first extraction unit 4030 is further configured to: set the stroke first sampling point identifier to a third value if the sampling point preceding the sampling point is an interrupt sampling point; and set the stroke first sampling point identifier to a fourth value if the sampling point preceding the sampling point is not an interrupt sampling point.

[0688] Optionally, the temporal spatial feature extraction model includes a long short-term memory network and a convolutional layer;

[0689] The first extraction unit 4030 is further specifically configured to:

[0690] Input the sampling point features into the long short-term memory network to obtain the first intermediate vector;

[0691] The first intermediate vector is input into the convolutional layer to obtain the temporal spatial features.

[0692] Optionally, the temporal spatial feature extraction model further includes a first linear layer and a maximum pooling layer;

[0693] The first extraction unit 4030 is further specifically configured to:

[0694] Input the first intermediate vector into the first linear layer to obtain the second intermediate vector;

[0695] Input the second intermediate vector into the convolution layer to obtain the third intermediate vector;

[0696] The third intermediate vector is input into the maximum pooling layer to obtain the temporal spatial features.

[0697] Optionally, the first input unit 4050 is specifically configured to:

[0698] Determine the embedding vector corresponding to the above information;

[0699] The position encoding information of each character in the above information is added to the embedding vector;

[0700] The embedding vector and temporal spatial features are input into the fusion prediction model to obtain the predicted text information.

[0701] Optionally, the fusion prediction model includes a multi-head attention model and a fusion layer;

[0702] The first input unit 4050 is further configured to:

[0703] Input the embedding vector into the multi-head attention model to obtain the attention vector;

[0704] Input the attention vector and temporal spatial features into the fusion layer to obtain the fusion vector;

[0705] The fused vector is input into the decision layer to obtain the predicted text information.

[0706] Optionally, the first input unit 4050 is further configured to:

[0707] Superimpose the attention vector and the embedding vector to obtain the first superposition vector;

[0708] Normalizing the first superposition vector to obtain a first normalized vector;

[0709] The first normalized vector and the temporal spatial features are input into the fusion layer to obtain the fused vector.

[0710] Optionally, the first input unit 4050 is further configured to:

[0711] Input the first normalized vector and the temporal spatial feature into the fusion layer to obtain a fourth intermediate vector;

[0712] Superimposing the fourth intermediate vector, the first normalized vector, and the temporal spatial feature to obtain a second superimposed vector;

[0713] Normalizing the second superposition vector to obtain a second normalized vector;

[0714] Input the second normalized vector into the second linear layer to obtain a fifth intermediate vector;

[0715] Superimposing the second normalized vector and the fifth intermediate vector to obtain a third superimposed vector;

[0716] Normalize the third superposition vector to obtain a fusion vector.

[0717] Optionally, the first input unit 4050 is further configured to:

[0718] Input the fused vector into the third linear layer to obtain the sixth intermediate vector;

[0719] Inputting the sixth intermediate vector into the decision layer to obtain the probabilities of multiple candidate text information;

[0720] Based on the probabilities of the plurality of candidate text information, the predicted text information is determined.

[0721] Reference Figure 41 , Figure 41 The structure block diagram of the terminal for implementing the handwritten text prediction method according to the embodiment of the present disclosure is as follows: the terminal includes: a radio frequency (RF) circuit 4110, a memory 4115, an input unit 4130, a display unit 4140, a sensor 4150, an audio circuit 4160, a wireless fidelity (WiFi) module 4170, a processor 4180, and a power supply 4190. Those skilled in the art will understand that Figure 41 The terminal structure shown does not constitute a limitation on the mobile phone or computer, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0722] The RF circuit 4110 may be used for receiving and sending signals during information transmission or calls. In particular, after receiving downlink information from the base station, it is sent to the processor 4180 for processing. In addition, the designed uplink data is sent to the base station.

[0723] The memory 4115 may be used to store software programs and modules. The processor 4180 executes various functional applications and data processing of the content terminal by running the software programs and modules stored in the memory 4115 .

[0724] The input unit 4130 may be configured to receive input digital or character information and generate key signal input related to the settings and function control of the content terminal. Specifically, the input unit 4130 may include a touch panel 4131 and other input devices 4132.

[0725] The display unit 4140 may be configured to display input information or provided information and various menus of the content terminal. The display unit 4140 may include a display panel 4141.

[0726] The audio circuit 4160 , the speaker 4161 , and the microphone 4162 may provide an audio interface.

[0727] In this embodiment, the processor 4180 included in the terminal can execute the handwritten text prediction method of the previous embodiment.

[0728] The terminals of the embodiments of the present disclosure include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc. The embodiments of the present invention can be applied to various scenarios, including but not limited to content recommendation, data screening, etc.

[0729] Figure 42 This is a block diagram of the structure of a portion of server 140 that implements the handwritten text prediction method according to an embodiment of the present disclosure. Server 140 may vary significantly due to different configurations or performance, and may include one or more central processing units (CPUs) 4222 (e.g., one or more processors), memory 4232, and one or more storage media 4230 (e.g., one or more mass storage devices) that store application programs 4242 or data 4244. Memory 4232 and storage media 4230 may be either transient or persistent storage. The program stored in storage medium 4230 may include one or more modules (not shown), each of which may include a series of instruction operations on server 140. Furthermore, CPU 4222 may be configured to communicate with storage medium 4230 to execute the series of instruction operations in storage medium 4230 on server 140.

[0730] The server 140 may also include one or more power supplies 4226, one or more wired or wireless network interfaces 4250, one or more input and output interfaces 4258, and / or one or more operating systems 4241, such as Wi ndows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0731] The central processing unit 4222 in the server 140 may be configured to execute the handwritten text prediction method according to the embodiment of the present disclosure.

[0732] The embodiments of the present disclosure further provide a computer-readable storage medium, which is used to store program codes, and the program codes are used to execute the handwritten text prediction methods of the aforementioned embodiments.

[0733] The present disclosure also provides a computer program product, which includes a computer program. A processor of a computer device reads and executes the computer program, so that the computer device implements the above-mentioned handwritten text prediction.

[0734] The terms "first," "second," "third," "fourth," and the like (if any) in the specification of the present disclosure and the accompanying drawings are used to distinguish between similar contents and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present disclosure described herein, for example, can be implemented in orders other than those illustrated or described herein. In addition, the terms "comprises" and "comprising," and any variations thereof, are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0735] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated content, indicating that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and following associated content is in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0736] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to exclude the number itself, and above, below, within, etc. are understood to include the number itself.

[0737] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0738] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0739] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0740] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server 140, or network device, etc.) to execute all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program code.

[0741] It should also be understood that the various implementations provided in the embodiments of the present disclosure can be combined arbitrarily to achieve different technical effects.

[0742] The above is a specific description of the implementation methods of the present disclosure, but the present disclosure is not limited to the above implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present disclosure. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. A handwritten text prediction method, characterized in that: include: Get the target handwritten text trajectory; Sampling the target handwritten text trajectory in time sequence to obtain time series sampling coordinates; Extracting temporal spatial features of the target handwritten text trajectory from the temporal sampling coordinates; Acquire context information of the target handwritten text track from a recognition result of a handwritten text input before the target handwritten text track; The temporal spatial features and the above information are input into a fusion prediction model to obtain predicted text information.

2. The handwritten text prediction method according to claim 1, characterized in that: The time series sampling coordinates are the coordinates of multiple sampling points in time sequence; The extracting the temporal spatial features of the target handwritten text trajectory from the temporal sampling coordinates includes: Based on the time series sampling coordinates, acquiring sampling point features of the target handwritten text trajectory; The sampling point features are input into a temporal spatial feature extraction model to obtain the temporal spatial features.

3. The handwritten text prediction method according to claim 2, characterized in that: The temporal spatial feature extraction model and the fusion prediction model are jointly trained through the following process: Acquire a set of sample handwritten text trajectories, wherein the sample handwritten text trajectories in the set of sample handwritten text trajectories have a sample previous context label and a sample next context label; Sampling the sample handwritten text trajectory in time sequence to obtain sample time series sampling coordinates; Extracting sample temporal spatial features of the sample handwritten text trajectory from the sample temporal sampling coordinates; Inputting the sample temporal spatial features and the sample context labels into a fusion prediction model to obtain a predicted sample context; Based on the difference between the predicted sample context and the sample context label, a loss function is calculated, and the temporal spatial feature extraction model and the fusion prediction model are jointly trained using the loss function.

4. The handwritten text prediction method according to claim 3, characterized in that: After obtaining the sample handwritten text trajectory set, the joint training process of the temporal spatial feature extraction model and the fusion prediction model further includes: Selecting adjacent stroke pairs to be exchanged from the sample handwritten text trajectories in the sample handwritten text trajectory set; Interchanging the adjacent strokes in the adjacent stroke pairs to be interchanged to obtain a handwritten text trajectory after the interchange; The swapped handwritten text trajectory is expanded into the set of sample handwritten text trajectories, wherein the swapped handwritten text trajectory has the same sample upper-context label and the same sample lower-context label as the sample handwritten text trajectory.

5. The handwritten text prediction method according to claim 3, characterized in that: After obtaining the sample handwritten text trajectory set, the joint training process of the temporal spatial feature extraction model and the fusion prediction model further includes: Rotating the sample handwritten text trajectories in the sample handwritten text trajectory set by multiple rotation angles to obtain multiple rotated handwritten text trajectories, wherein the multiple rotation angles are in an arithmetic progression; A plurality of rotated handwritten text trajectories are added to the sample handwritten text trajectory set, wherein the rotated handwritten text trajectories have the same sample upper-context labels and the same sample lower-context labels as the sample handwritten text trajectories.

6. The handwritten text prediction method according to claim 3, characterized in that: The sampling of the sample handwritten text trajectory in time sequence to obtain sample time sequence sampling coordinates includes: sampling the sample handwritten text trajectory in time sequence according to multiple sampling frequencies to obtain multiple groups of sample time sequence sampling coordinates; The step of inputting the sample temporal spatial features and the sample context labels into a fusion prediction model to obtain a predicted sample context includes: for each group of sample temporal sampling coordinates, inputting the sample temporal spatial features and the sample context labels corresponding to the group of sample temporal sampling coordinates into the fusion prediction model to obtain the sample context predicted for the group of sample temporal sampling coordinates; The calculating of the loss function based on the difference between the predicted sample context and the sample context label includes: calculating the loss function corresponding to the group of sample time series sampling coordinates based on the difference between the sample context predicted for the group of sample time series sampling coordinates and the sample context label.

7. The handwritten text prediction method according to claim 2, wherein: The acquiring, based on the time series sampling coordinates, sampling point features of the target handwritten text trajectory includes: Determining a normalized displacement difference of the sampling point based on a first coordinate of the sampling point and a second coordinate of a sampling point preceding the sampling point; Obtaining the pen lift and drop identifier of the sampling point; Obtaining a stroke first sampling point identifier, where the stroke first sampling point identifier indicates whether the sampling point is the first sampling point in the stroke where the sampling point is located; The sampling point features of the target handwritten text trajectory are generated based on the normalized displacement difference, the pen lift and pen drop identifiers, and the stroke first sampling point identifier.

8. The handwritten text prediction method according to claim 7, characterized in that: The first coordinate includes a first abscissa and a second ordinate, and the second coordinate includes a second abscissa and a second ordinate; The determining the normalized displacement difference of the sampling point based on the first coordinate of the sampling point and the second coordinate of a sampling point preceding the sampling point includes: Determine a first difference between the first horizontal coordinate of the sampling point and the second horizontal coordinate of a sampling point preceding the sampling point, and a second difference between the first vertical coordinate of the sampling point and the second vertical coordinate of the sampling point preceding the sampling point; Obtaining a maximum value and a minimum value of the ordinate of the target handwritten text trajectory; Determining a vertical coordinate span based on the vertical coordinate maximum value and the vertical coordinate minimum value; Determining a normalized horizontal displacement differential of the sampling point based on the first difference and the vertical coordinate span, and determining a normalized vertical displacement differential of the sampling point based on the second difference and the vertical coordinate span; Based on the normalized horizontal coordinate displacement difference and the normalized vertical coordinate displacement difference, the normalized displacement difference of the sampling point is determined.

9. The handwritten text prediction method according to claim 7, wherein: The step of obtaining the pen lift and drop identifier of the sampling point includes: Acquire a first pressure at the sampling point and a second pressure at a sampling point preceding the sampling point; If the first pressure is less than the second pressure, determining the pen-lifting / lowering flag as a first value indicating pen-lifting; If the first pressure is not less than the second pressure, the pen-up / pen-down flag is determined as a second value indicating pen-down.

10. The handwritten text prediction method according to claim 7, characterized in that: Before determining the normalized displacement difference of the sampling point based on the first coordinate of the sampling point and the second coordinate of a sampling point preceding the sampling point, the handwritten text prediction method further includes: Setting an interruption sampling point at a position where a stroke of the target handwritten text trajectory is interrupted, wherein the abscissa of the interruption sampling point is outside a value range of the abscissa of the target handwritten text trajectory, and the ordinate of the interruption sampling point is outside a value range of the ordinate of the target handwritten text trajectory; adding the interrupted sampling point to the plurality of sampling points; The obtaining of the stroke first sampling point identifier includes: if the sampling point preceding the sampling point is the interruption sampling point, setting the stroke first sampling point identifier to a third value; if the sampling point preceding the sampling point is not the interruption sampling point, setting the stroke first sampling point identifier to a fourth value.

11. The handwritten text prediction method according to claim 2, wherein: The temporal spatial feature extraction model includes a long short-term memory network and a convolutional layer; Inputting the sampling point features into a temporal-spatial feature extraction model to obtain the temporal-spatial features includes: Inputting the sampling point features into the long short-term memory network to obtain a first intermediate vector; The first intermediate vector is input into the convolutional layer to obtain the temporal spatial feature.

12. The handwritten text prediction method according to claim 11, characterized in that: The temporal spatial feature extraction model further includes a first linear layer and a maximum pooling layer; Inputting the first intermediate vector into the convolutional layer to obtain the temporal spatial feature includes: Inputting the first intermediate vector into the first linear layer to obtain a second intermediate vector; Inputting the second intermediate vector into the convolutional layer to obtain a third intermediate vector; The third intermediate vector is input into the maximum pooling layer to obtain temporal spatial features.

13. The handwritten text prediction method according to claim 1, wherein: The step of inputting the temporal spatial features and the above context information into a fusion prediction model to obtain predicted text information includes: Determine an embedding vector corresponding to the above information; Adding the position encoding information of each character of the above information to the embedding vector; The embedding vector and the temporal spatial feature are input into the fusion prediction model to obtain the predicted text information.

14. The handwritten text prediction method according to claim 13, wherein: The fusion prediction model includes a multi-head attention model and a fusion layer; Inputting the embedding vector and the temporal spatial feature into the fusion prediction model to obtain the predicted text information includes: Inputting the embedding vector into the multi-head attention model to obtain an attention vector; Inputting the attention vector and the temporal spatial feature into a fusion layer to obtain a fusion vector; The fusion vector is input into the decision layer to obtain the predicted text information.

15. The handwritten text prediction method according to claim 14, characterized in that: The step of inputting the attention vector and the temporal spatial feature into a fusion layer to obtain a fusion vector includes: Superimposing the attention vector and the embedding vector to obtain a first superimposed vector; Normalizing the first superposition vector to obtain a first normalized vector; The first normalized vector and the temporal spatial feature are input into the fusion layer to obtain the fusion vector.

16. The handwritten text prediction method according to claim 15, characterized in that: Inputting the first normalized vector and the temporal spatial feature into the fusion layer to obtain the fusion vector includes: Inputting the first normalized vector and the temporal spatial feature into the fusion layer to obtain a fourth intermediate vector; Superimposing the fourth intermediate vector, the first normalized vector, and the temporal spatial feature to obtain a second superimposed vector; Normalizing the second superposition vector to obtain a second normalized vector; Inputting the second normalized vector into the second linear layer to obtain a fifth intermediate vector; Superimposing the second normalized vector and the fifth intermediate vector to obtain a third superimposed vector; The third superposition vector is normalized to obtain the fusion vector.

17. A handwritten text prediction device, characterized in that: include: A first acquisition unit, configured to acquire a target handwritten text trajectory; A first sampling unit is used to sample the target handwritten text trajectory in time sequence to obtain time series sampling coordinates; A first extraction unit, configured to extract temporal spatial features of the target handwritten text trajectory from the temporal sampling coordinates; A second acquiring unit, configured to acquire context information of the target handwritten text trajectory from a recognition result of a handwritten text input before the target handwritten text trajectory; The first input unit is used to input the temporal spatial features and the above information into a fusion prediction model to obtain predicted text information.

18. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the handwritten text prediction method according to any one of claims 1 to 16 is implemented.

19. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the handwritten text prediction method according to any one of claims 1 to 16 is implemented.

20. A computer program product, comprising a computer program, wherein the computer program is read and executed by a processor of a computer device, so that the computer device executes the handwritten text prediction method according to any one of claims 1 to 16.

Images