Mathematical formula handwriting trajectory recognition method and system based on double model cooperation and local adaptive learning

Abstract

The application discloses a mathematical formula handwriting trajectory recognition method and system based on double-model cooperation and local adaptive learning. The method comprises the following steps: obtaining a handwriting vector trajectory sequence; inputting the sequence into a general trajectory recognition model and a local personalized adaptation model based on LoRA for parallel inference; outputting a recognition result according to a confidence degree; incrementally updating the personalized adaptation model locally by using user interaction confirmation or correction behavior; and creatively introducing a space structure consistency regularization term based on KL divergence during the updating process, so as to constrain the change amplitude of the space structure label prediction distribution of the decoder, thereby preventing the personalized adaptation from damaging the two-dimensional structure analysis capability. The application also identifies a writing mode switching event through handwriting dynamic characteristics, triggers an auxiliary operation, and adopts a double-condition triggered dynamic loading and early stopping mechanism to perform power adaptive scheduling, so as to guarantee the low-delay real-time performance of a mobile terminal and realize double-model cooperative inference. The application realizes continuous and unobtrusive adaptation of individual writing styles without leaking original handwriting of a user, and significantly improves the personalized recognition accuracy and interaction experience of handwritten mathematical formulas.

Description

Technical Field

[0001] This invention belongs to the field of pattern recognition and human-computer interaction technology, specifically relating to an online method and system for recognizing mathematical formulas based on handwriting pen vector trajectories. It is particularly suitable for educational technology, collaborative whiteboards, and computing devices that require high-precision real-time interaction and can adapt to individual handwriting styles. Background Technology

[0002] Real-time handwritten recognition of mathematical formulas is a key technical challenge in fields such as educational informatization, online collaboration, and scientific computing. Mathematical formulas have complex two-dimensional structures (fractions, square roots, subscripts and superscripts, matrices, etc.), and the recognition system must simultaneously capture the symbol category and spatial layout, making it far more technically difficult than one-dimensional text recognition.

[0003] The current mainstream solutions have the following significant drawbacks:

[0004] First, general-purpose models cannot cope with individual differences. While existing general-purpose recognition models based on deep learning have high average accuracy, they lack the ability to adapt to unique individual writing habits (such as stroke order, cursive writing, and personalized symbol deformation). When a user's writing style deviates significantly from the distribution of training data, the recognition accuracy drops sharply, severely impairing the user experience.

[0005] Second, personalized solutions are inefficient and compromise privacy. Traditional personalization methods require users to provide a large number of labeled samples and upload them to the cloud for retraining, which is not only cumbersome and unable to take effect in real time, but also poses a serious risk of handwriting privacy leakage. This conflict between privacy and personalization has long remained unresolved.

[0006] Third, separating structural information from handwriting style is difficult. Recognizing mathematical formulas requires not only identifying the symbols themselves but also accurately reconstructing their two-dimensional spatial relationships. Existing solutions, when adapting to individual handwriting styles, often neglect personalized modeling of spatial structures (such as fraction bar length, square root coverage, and subscript / superscript offsets). Furthermore, while lightweight fine-tuning techniques like LoRA have been applied in natural language processing, directly applying them to mathematical formula recognition can lead to increased structural ambiguity in long formulas and complex nested structures, and even catastrophic forgetting of structural marker predictions, because LoRA only adapts to token probabilities at the symbol level and ignores adaptation to two-dimensional spatial structures.

[0007] Fourth, handwriting behavior information is not being utilized. The dynamic characteristics of a user's writing process, such as pen speed, pauses, pen lifting frequency, and retracing of specific areas, contain rich information about the user's interactive state. Existing recognition systems completely ignore these features and cannot perceive changes in the user's behavioral patterns during interaction.

[0008] Fifth, low-computing-power terminals lack real-time performance. The inference computation of a complete Transformer model is large. Running dual models in real time on mobile terminals such as tablets will face a computing power bottleneck, resulting in excessive recognition latency and affecting the smooth experience of "writing and converting simultaneously".

[0009] Therefore, there is an urgent need for an online mathematical formula recognition solution that can safely, in real time, and continuously adapt to individual writing habits, while making full use of dynamic handwriting features and ensuring low latency. Summary of the Invention

[0010] This invention aims to provide a method and system for recognizing handwritten mathematical formula trajectories based on dual-model collaboration and local adaptive learning, in order to solve the following technical problems:

[0011] First, how to achieve real-time, seamless, and continuous adaptation to individual handwriting styles without revealing the user's original handwriting, significantly improving personalized recognition accuracy; second, how to adapt to individual handwriting styles while preventing damage to the ability to analyze two-dimensional spatial structures, avoiding structural ambiguity or catastrophic forgetting due to personalized adaptation; third, how to effectively utilize the dynamic behavioral characteristics of handwriting to perceive changes in interactive states and trigger corresponding system responses; fourth, how to efficiently schedule dual models on low-computing-power terminals to achieve low-latency real-time recognition while ensuring recognition quality.

[0012] To address the aforementioned technical problems, this invention provides a method for recognizing handwritten mathematical formula trajectories based on dual-model collaboration and local adaptive learning, comprising the following steps:

[0013] Step 1: Acquisition and preprocessing of handwriting vector trajectory data. Vector trajectory data of pen tip movement is acquired in real time using a handwriting input device, forming a point sequence with timestamps and pressure: P = {(x_i, y_i, t_i, p_i) | i =1, 2, ..., N}. The point sequence is then denoised, smoothed, and resampled, and the kinematic characteristics of each sampled point are calculated: instantaneous velocity, acceleration, direction angle, etc., to construct an enhanced vector trajectory sequence.

[0014] Optionally, before inputting the enhanced vector trajectory sequence into the recognition model, the model further includes stroke segmentation and atomic stroke pre-classification of the handwriting to generate a compact set of symbol candidates to reduce the decoding search space of the subsequent model.

[0015] Step two involves parallel inference using a dual-model approach based on a general model and a personalized adaptation model. The enhanced vector trajectory sequence is input into both the general trajectory recognition model and the personalized adaptation model. The general trajectory recognition model, based on a Transformer encoder-decoder architecture and pre-trained with large-scale multi-user handwritten mathematical formula data, is used to recognize symbol categories and two-dimensional spatial structures, outputting a first recognition result and a first confidence score. The personalized adaptation model, built upon the general trajectory recognition model, is constructed by inserting trainable adapter parameters into the attention and feedforward layers of the Transformer network using low-rank adaptation (LoRA) technology. Initially, the adapter parameters are set to zero, and the model output is equivalent to the general model. This model outputs a second recognition result and a second confidence score. All parameters of the personalized adaptation model are stored and updated locally on the user's device and do not leave the user's device.

[0016] Step 3, Dynamic Routing Decision. Based on a preset routing strategy, the first and second identification results are fused and a final identification result is output. The routing strategy is based on at least the following rules: when the second confidence level exceeds the first preset threshold, the second identification result is adopted; when the second confidence level is lower than the second preset threshold and the first confidence level is higher than the second confidence level, the first identification result is adopted; when both confidence levels are lower than an acceptable threshold, user interaction confirmation is triggered.

[0017] Step four: Local incremental update based on user interaction confirmation or correction behavior. When a user interaction confirmation or correction behavior is received regarding the final recognition result (the interaction confirmation or correction behavior includes, but is not limited to: the user selecting the correct candidate in the candidate list, the user clicking on the incorrectly recognized area and entering the correct symbol, the user crossing out the incorrect character and rewriting it via gesture), the enhanced vector trajectory sequence and the label corresponding to the feedback are used to form a training sample pair, which is temporarily stored in the local cache. When the number of cached samples reaches the preset batch size, the parameters of the general trajectory recognition model are frozen on the local device, and only the LoRA adapter parameters of the personalized adaptation model are updated once.

[0018] The gradient update loss function creatively introduces a spatial structure consistency regularization term. This regularization term constrains the change in probability distribution by calculating the KL divergence of the output probability distribution of spatial structure markers (such as the position of fractional horizontal lines, the coverage of square roots, and the offset of superscripts and subscripts) in the same input before and after the adapter update. Thus, personalized adaptation not only learns individual habits of symbol shape but also robustly learns individual typography styles, while preventing the adaptation process from destroying the two-dimensional structure resolution capability already possessed by the general model. This overcomes the catastrophic structural forgetting problem that may occur when ordinary LoRA is directly applied to mathematical formula recognition. The update operation is executed silently in a background thread, without affecting the user's current writing operation. When a user interaction confirmation is triggered in step three due to low confidence, this confirmation operation simultaneously enters the incremental update process of this step as an interactive confirmation or correction behavior, forming a complete closed loop of recognition-feedback-optimization.

[0019] Step 5: Extraction of dynamic handwriting features and triggering of writing mode switching events. Dynamic interaction features are extracted from the vector trajectory sequence, including at least: writing speed sequence, pause time between strokes, frequency of retracing in local areas, and pressure fluctuation parameters. These dynamic interaction features are then clustered or classified online to generate a status label for the current writing interval. The status label at least represents a first writing mode (low speed, high pause) and a second writing mode (high speed, low pause). When the system detects that the status label switches from the first writing mode to the second writing mode within a preset time window, and simultaneously the confidence level shows a step increase, it is determined as a writing mode switching event, triggering the system to automatically execute preset auxiliary operations, such as automatically saving a snapshot of the current writing content or activating the auxiliary analysis interface.

[0020] Step Six: Adaptive Computing Power Scheduling. To meet the low latency requirements of mobile terminals, the dual-model parallel inference employs an adaptive computing power scheduling strategy combining dynamic loading and early stopping. The loading strategy uses a dual-condition triggering logic: the system loads and activates the personalized adaptation model when either of the following conditions is met: (a) the average recognition confidence within the current sliding window remains below a preset confidence trigger threshold; (b) the frequency of user interaction confirmation or correction for a specific symbol category exceeds a preset frequency trigger threshold within the sliding window. Furthermore, to ensure a good initial user experience, the system loads the personalized adaptation model by default after the user writes the first M symbols (M is a preset positive integer, such as 5) to collect initial adaptation data. The early stopping strategy stipulates that when the first confidence generated by the general trajectory recognition model exceeds a preset extremely high threshold, the inference process of the personalized adaptation model is skipped directly, and only the first recognition result is output to reduce computational overhead.

[0021] Compared with the prior art, the present invention has the following significant advantages:

[0022] First, local, seamless adaptive learning ensures user handwriting privacy. Updates to the personalized adaptation model are completed entirely on the user's device in a closed loop, with original handwriting data never uploaded. Users require no manual training; recognition rates continuously improve during natural writing, completely resolving the conflict between privacy and personalization.

[0023] Second, robust adaptation under spatial structure probabilistic constraints leads to more accurate and stable recognition. By specifically introducing a spatial structure consistency regularization term based on KL divergence into the LoRA update objective, personalized adaptation can not only learn "symbol shapes" but also robustly learn "personal typography habits." This regularization term effectively constrains the decoder's prediction distribution of spatial structure markers (fraction lines, square root ranges, subscript and superscript offsets) from drastically shifting due to adaptation, thus overcoming the catastrophic structural forgetting problem that may occur when ordinary LoRA is directly applied to mathematical formula recognition. This is a creative improvement specifically for the field of mathematical formula recognition.

[0024] Third, the detection and utilization of writing mode switching opens up a new dimension of human-computer interaction. By extracting dynamic interaction features from the handwritten trajectory of mathematical formulas and determining mode switching events, the system can automatically execute response operations based on the leap in writing fluency, such as automatically saving when the user's thoughts are flowing smoothly, triggering auxiliary tools, etc., thus building an intelligent system that better understands the rhythm of user interaction.

[0025] Fourth, efficient computing power scheduling ensures real-time performance on mobile devices. By using dynamic loading triggered by dual conditions and an early stopping strategy driven by confidence, the average computing power consumption of the dual-model architecture is significantly reduced. At the same time, the default loading strategy for cold starts ensures a good initial experience for new users, making smooth "write-while-transfer" a reality on mobile terminals such as tablets. Attached Figure Description

[0026] Figure 1 This is an overall flowchart of the method of the present invention.

[0027] Figure 2 This is a schematic diagram of the architecture for dual-model inference and dynamic routing decision-making.

[0028] Figure 3 The flowchart shows the local incremental update process for the personalized adaptation model, with the calculation location of the spatial structure consistency regularization term specifically marked in the diagram.

[0029] Figure 4 This is a timing diagram illustrating the dynamic feature extraction of handwriting and the triggering of writing mode switching events.

[0030] Figure 5 This is the state transition diagram for the computing power adaptive scheduling strategy. Detailed Implementation

[0031] The present invention will now be described in detail with reference to the embodiments and accompanying drawings. It should be noted that the following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0032] I. Hardware Environment and Data Acquisition.

[0033] This embodiment uses a tablet computer equipped with an active stylus as the hardware platform, with a handwriting sampling rate of 120Hz. When the user writes, the system driver layer reports the touch event in real time, forming an original vector sequence: S_raw = {(x_i, y_i,t_i, p_i)}_{i=1}^{N}, where (x_i, y_i) are the coordinates normalized to the standard writing area, t_i is the millisecond-level timestamp, and p_i is the normalized pressure value (between 0 and 1).

[0034] After low-pass filtering to remove noise, the original sequence is segmented into independent strokes based on the stroke lifting event. The first-order difference of each sampling point is then calculated to obtain the velocity v_i, acceleration a_i, and orientation angle theta_i = arctan2(Deltay, Delta x), forming the enhanced vector sequence S_enh, which serves as the unified input for all subsequent models.

[0035] Optionally, each stroke in S_enh can be classified into atomic strokes (categorized into basic stroke types such as horizontal, vertical, diagonal, curved, and circular), and pre-screened based on the spatial relationships between strokes to generate a compact set of candidate symbols. This candidate set will be injected as prior knowledge into the subsequent decoding process, effectively reducing the search space.

[0036] II. General trajectory recognition model.

[0037] The general trajectory recognition model employs a Transformer network with an encoder-decoder structure. The encoder maps S_enh to a sequence of hidden representations. To enable the model to perceive two-dimensional spatial structure, learnable two-dimensional positional encodings are added to the input embedding layer, encoding the horizontal and vertical coordinates separately and then fusing them, allowing the model to distinguish the spatial relationships of symbols on the writing plane. The decoder generates the target LaTeX sequence symbol by symbol using an autoregressive approach. During decoding, the model needs to correctly predict two types of markers: symbol markers (such as x, +, \int) and structure markers (such as \frac{}{}, \sqrt{}, ^{}, _{}). The correctness of the prediction of structure markers directly determines whether the two-dimensional structure of the output formula is valid. This model is pre-trained on a massive dataset of handwritten mathematical formulas from multiple users and is deployed as a general-purpose platform to user terminals.

[0038] III. Personalized Adaptation Model and Local Incremental Updates.

[0039] The personalized adaptation model shares the same master parameters as the general model, but maintains an independent set of low-rank adapter parameters Θ_lora. The LoRA adapter is inserted at the bypass of the attention weight matrix (Query, Key, Value projection) and part of the feedforward layer weight matrix of each layer of the Transformer.

[0040] Initialization: When a user uses the model for the first time, Θ_lora is set to zero, and the personalized model output is completely consistent with the general model.

[0041] Spatial structure consistency regularization term: This is one of the core innovations of this invention. During each local incremental update, the loss function consists of two parts: L_total = L_recog + λ · L_struct, where L_recog is the standard recognition cross-entropy loss, L_struct is the spatial structure consistency regularization term, and λ is the balance coefficient.

[0042] The specific calculation method of L_struct is as follows: Before and after the adapter update, forward inference is performed on the same input using the current model and the model before the update, respectively, to obtain the output probability distribution of the decoder for structure class labels (such as \frac, \sqrt, ^, _) at each time step. The KL divergence between these two probability distributions is calculated: L_struct = Σ_{t∈T_struct}D_KL(P_t^{old} ∥ P_t^{new}), where T_struct is the set of time steps for all corresponding structure class labels in the output sequence. This regularization term effectively constrains the extent of modification of the personalized adaptation to the two-dimensional structure parsing logic—the user's personalized writing habits (such as writing particularly long fraction lines) will be learned by the LoRA adapter at the symbol shape level, but for structural decision logic such as "when should a fraction structure appear, and how should the numerator and denominator of the fraction be divided", the system remains robust and avoids degradation due to adaptation.

[0043] Local Incremental Update Process: When a user provides feedback through interactive confirmation or correction (e.g., selecting the correct symbol in the candidate list, clicking on an incorrect area and entering the correct symbol, or crossing out an incorrect symbol and rewriting), the system stores the corresponding label and handwriting vector fragment in a local cache. When the cache accumulates to the batch size (e.g., 16 entries), a parameter update is performed in a background thread. The optimizer uses AdamW with a learning rate of 1×10^{-4}. The cache is cleared after the update is complete. Throughout the process, all training data and model parameters are stored and processed locally only, without network transmission. The update operation is executed silently in a background thread and does not affect the user's current writing operation. When a user interaction confirmation is triggered in step three due to low confidence, this confirmation operation also enters the incremental update process of this step as an interactive confirmation or correction behavior, forming a complete closed loop of recognition-feedback-optimization.

[0044] IV. Dynamic routing decision-making.

[0045] After parallel inference by the two models, symbol sequences and confidence scores are obtained respectively. In this embodiment, the confidence score is the negative entropy normalized value of the decoder output probability distribution, with a value range of [0,1]. The routing rules are set with a high threshold of 0.9 and a low threshold of 0.6. At the same time, a user-level "symbol preference table" is maintained to record the mapping of specific symbol variants that a user has confirmed multiple times in history (for example, a user's specific trajectory pattern is repeatedly confirmed as the handwritten "z" instead of the number "2"). When the second confidence score exceeds 0.9, the second recognition result is directly adopted; when the second confidence score is lower than 0.6 and the first confidence score is higher, the first recognition result is adopted. In marginal cases, the symbol preference table is used to assist in the judgment. If both confidence scores are lower than 0.5, the system highlights the uncertain area on the interface and requests the user to click to confirm. This confirmation operation is also used as an interactive confirmation or correction behavior to enter the incremental update process in step four.

[0046] V. Handwriting dynamic feature extraction and writing mode switching events.

[0047] The system continuously collects dynamic user characteristics in the background: average writing speed for each symbol, pause time between symbols, number of retracings in a local area (within a radius of 50 pixels), and pressure standard deviation. These characteristics are input into a lightweight online clustering module, which labels the current writing interval as either mode 1 (low speed, high pause) or mode 2 (high speed, low pause) in real time. When mode 1 is continuously detected for at least k symbols (e.g., 5 symbols), followed by mode 2 for m symbols (e.g., 3 symbols), and the confidence level increases dramatically (e.g., from below 0.6 to above 0.9), it is considered a writing mode switching event. The application layer can then automatically trigger functions such as note saving, mind map expansion, or auxiliary interface switching to match the user's interaction rhythm.

[0048] VI. Adaptive scheduling of computing power.

[0049] The computing power scheduling strategy in this embodiment includes three parts: a cold start strategy, a dynamic loading strategy, and an early stop strategy. Cold start strategy: When a new user uses the system for the first time, the system loads the personalized adaptation model by default after the user writes the first 5 symbols, allowing it to start collecting adaptation data early and avoiding a "cold start deadlock" due to poor recognition by the general model and lack of adaptation trigger conditions. Dynamic loading strategy: In daily use, only the general model runs by default. When any of the following conditions are met, the personalized adaptation model parameters are asynchronously loaded in the background, and dual-model inference is started: (a) the average recognition confidence within the current sliding window (the most recent 20 symbols) remains below 0.7; (b) the user's interaction confirmation or correction behavior for a specific symbol category exceeds 3 times within the sliding window. Early stop strategy: In dual-model inference mode, if the general model's confidence in the current handwriting exceeds the extremely high threshold of 0.98, the personalized adaptation model inference is skipped, and only the first recognition result is output. This three-level scheduling strategy significantly reduces the average CPU / GPU usage and power consumption while ensuring recognition quality.

Claims

1. A method for recognizing handwritten mathematical formula trajectories based on dual-model collaboration and local adaptive learning, characterized in that, include: Obtain the sequence of handwriting vector trajectories generated by handwriting input; The handwriting vector trajectory sequence is input into a general trajectory recognition model and a personalized adaptation model for parallel inference to obtain a first recognition result and a first confidence level, a second recognition result and a second confidence level; wherein the personalized adaptation model is constructed by inserting locally stored low-rank adapter parameters based on the general trajectory recognition model. Based on preset routing rules, the final identification result is output according to the first confidence level and the second confidence level; When a user's interactive confirmation or correction of the final recognition result is received, the corresponding label is used to form a training sample with the handwriting vector trajectory sequence. The parameters of the general trajectory recognition model are frozen on the local device, and only the low-rank adapter parameters of the personalized adaptation model are incrementally updated.

2. The method according to claim 1, characterized in that, The incremental update loss function introduces a spatial structure consistency regularization term; this regularization term constrains the change range of the probability distribution of the decoder's output probability distribution of spatial structure labels in the same input before and after the adapter parameter update, so as to prevent personalized adaptation from destroying the ability to analyze the two-dimensional structure of mathematical formulas.

3. The method according to claim 1, characterized in that, The preset routing rules include at least the following: When the second confidence level exceeds the first preset threshold, the second identification result is adopted; When the second confidence level is lower than the second preset threshold and the first confidence level is higher than the second confidence level, the first identification result is adopted.

4. The method according to claim 1, characterized in that, Also includes: Dynamic interaction features are extracted from the handwriting vector trajectory sequence, and the dynamic interaction features include at least writing speed, pause time between strokes, local retracing frequency, and pressure fluctuation parameters. Based on the dynamic interaction features, identify the writing mode label corresponding to the current writing area; When the writing mode label is detected to switch from the first mode to the second mode within a preset time window, and the recognition confidence level increases dramatically, a preset auxiliary operation is triggered.

5. The method according to claim 1, characterized in that, It also includes the computing power scheduling step: The personalized adaptation model is loaded and activated when any of the following conditions are met: (a) the average recognition confidence level in the current sliding window is consistently lower than the preset confidence level trigger threshold; (b) the frequency of user interaction confirmation or correction behavior for a specific symbol category exceeds the preset frequency trigger threshold in the sliding window. When the first confidence level generated by the general trajectory recognition model exceeds a preset extremely high threshold, the inference process of the personalized adaptation model is skipped.

6. The method according to claim 5, characterized in that, First-time users will load the personalized adaptation model by default after writing the first M symbols, where M is a preset positive integer.

7. The method according to claim 1, characterized in that, Before inputting the handwriting vector trajectory sequence into the general trajectory recognition model and the personalized adaptation model, the method further includes: performing stroke segmentation and atomic stroke pre-classification on the handwriting vector trajectory sequence to generate a candidate symbol set, so as to reduce the decoding search space of the subsequent model.

8. The method according to any one of claims 1 to 7, characterized in that, The incremental update of the personalized adaptation model is executed locally in the background on the user device, and the original data of the handwriting vector trajectory sequence does not leave the user device.

9. A mathematical formula handwritten trajectory recognition system based on dual-model collaboration and local adaptive learning, characterized in that, include: The data acquisition module is used to acquire the sequence of handwriting vector trajectories generated by handwriting input; The parallel inference engine, with a built-in general trajectory recognition model and a personalized adaptation model, is used to perform parallel inference on the handwriting vector trajectory sequence to obtain a first recognition result and a first confidence level, a second recognition result and a second confidence level. The personalized adaptation model is built on the general trajectory recognition model by inserting locally stored low-rank adapter parameters, and the loss function of its low-rank adapter parameters during the update includes a spatial structure consistency regularization term, which is used to constrain the change range of the decoder's output probability distribution of spatial structure markers. A dynamic routing arbitrator is used to output the final identification result based on the first confidence level and the second confidence level, according to preset routing rules. The local adaptive update module is used to, when receiving user interaction confirmation or correction behavior for the final recognition result, use the label corresponding to the feedback and the handwriting vector trajectory sequence to form a training sample, freeze the parameters of the general trajectory recognition model on the local device, and only perform incremental updates on the low-rank adapter parameters of the personalized adaptation model.

10. The system according to claim 9, characterized in that, Also includes: The behavioral feature analysis module is used to extract dynamic interaction features from the handwriting vector trajectory sequence and generate writing mode labels, and to trigger auxiliary operations when a mode switch is detected and the confidence level increases by a step. The computing power scheduling module is used to execute a dynamic loading strategy triggered by two conditions and an early stopping strategy driven by confidence, so as to control computing overhead while ensuring recognition quality.

11. A smart handwriting device, characterized in that, Includes the system described in claim 9 or 10.

Images