Writing trace optimization system based on machine learning

Abstract

The invention provides a writing track optimization system based on machine learning, and relates to the technical field of machine learning and artificial intelligence, and the system comprises a data collection module which synchronously collects inertial measurement and visual track data, and generates standardized writing data through dynamic alignment and coordinate unification; the feature extraction and fusion module is used for extracting spatio-temporal features and fusing multi-source features through a cross-modal attention mechanism; the model building module is used for building a double-branch multi-task collaborative model, classifying branches to predict character categories, regression branches to generate a space-time coordinate sequence, and designing a multi-objective optimization function fusing cross entropy loss, trajectory distance error and shape constraint loss; and the trajectory generation and recognition module is based on a two-stage training strategy of dynamic division, optimizes the overall shape similarity of the trajectory in the first training stage, enhances the distance precision of trajectory points in the second training stage, and realizes high-precision trajectory reconstruction and character recognition. According to the method, the problem of writing jitter is effectively solved, and the track smoothness and the recognition robustness are improved.

Description

Technical Field

[0001] The present invention relates to the field of machine learning and artificial intelligence technology, and in particular to a writing trajectory optimization system based on machine learning. Background Art

[0002] In interactive writing scenarios such as electronic whiteboards, user writing trajectories are easily affected by factors such as hand jitter, sensor noise, and visual occlusion, resulting in significant jitter and distortion in the collected trajectory data. Traditional trajectory capture systems based on a single modality (such as pure vision or pure inertial sensing) have difficulty overcoming inherent defects: visual trajectories are easily affected by changes in perspective, lighting conditions, and occlusion. Especially in multi-person collaborative electronic whiteboard scenarios, hand occlusion often causes pen tip positioning drift; and while inertial measurement units (IMUs) can capture subtle movements, they suffer from cumulative errors and loss of absolute position. Although multimodal data fusion has been proposed to complement each other's strengths, existing methods still have shortcomings in terms of temporal alignment accuracy and cross-modal feature collaboration, making it difficult to strike a balance between trajectory smoothness and recognition robustness.

[0003] Existing trajectory optimization solutions typically use post-processing filtering (such as Kalman filtering) or independently trained recognition models, which have significant limitations: while filtering methods can suppress high-frequency noise, they can over-smooth writing details, resulting in the loss of stroke turning features; treating trajectory generation and character recognition as separate tasks ignores the inherent correlation between the two, which can easily lead to error accumulation. In addition, traditional models have difficulty simultaneously optimizing the local point accuracy of the trajectory (such as the pen tip coordinate error) and the global shape similarity (such as the fidelity of the stroke structure). In real-time writing scenarios on electronic whiteboards, it is often impossible to strike a balance between jitter suppression and maintaining writing intent. Therefore, an end-to-end learning framework that can collaboratively optimize trajectory reconstruction and character recognition is urgently needed to improve the accuracy and fluency of the electronic whiteboard writing experience. Summary of the Invention

[0004] In order to solve the technical problems mentioned in the current background technology, the purpose of the present invention is to provide a writing trajectory optimization system based on machine learning.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions: The writing trajectory optimization system based on machine learning includes: Data acquisition module: collects first writing data of the writing process based on a preset synchronization strategy, including inertial measurement data and visual trajectory data; preprocesses the first writing data to generate second writing data, wherein the preprocessing includes dynamic length alignment, coordinate system one, and dynamic visual compensation; Feature extraction and fusion module: extracts and fuses features of the second written data using a preset feature model to generate fusion features; the feature model is constructed based on a convolutional neural network and a long short-term memory network; Model construction module: Constructing a multi-task collaborative model, the multi-task collaborative model adopts a dual-branch architecture, and setting different loss functions based on the dual-branch architecture; training the multi-task collaborative model in two training phases based on dynamic partitioning, including optimizing the shape similarity of writing trajectories in the first training phase and enhancing the distance accuracy of writing trajectory points in the second training phase; Trajectory generation and recognition module: inputs the fusion features into the multi-task collaborative model to generate the probability distribution of character categories and the spatiotemporal coordinate sequence of the writing trajectory.

[0006] Furthermore, the feature model includes an input layer, a feature extraction layer, a time series modeling layer, a feature fusion layer, a feature enhancement layer and an output layer; dividing the second written data into a plurality of local segments through a preset sliding window; The input layer inputs the local segment; The feature extraction layer is composed of a convolutional neural network, including temporal convolution and feature dimensionality reduction; The temporal convolution uses two layers of 1D-CNN to capture the local spatiotemporal patterns of local segments to generate abstract features; The feature dimensionality reduction compresses the abstract features through the pooling layer to generate compressed features ; The temporal modeling layer is composed of a two-layer bidirectional long short-term memory network, based on the compression feature Generate the second time series feature .

[0007] Furthermore, the feature fusion layer combines the compressed features And the second timing characteristics Perform cross-modal feature fusion to generate the fusion feature. The specific steps are as follows: 1) The compression feature And the second timing characteristics Align according to the preset timeline; 2) Fusion of the compression features And the second timing characteristics Generate the first fusion feature:

[0008] in, is the time step t The first fusion feature when is the time step t The fusion weight when is element-wise multiplication; The fusion weight is obtained through the attention weight formula, which is as follows:

[0009] in, and is a learnable parameter, is the activation function; The feature enhancement layer normalizes the first fusion feature:

[0010] in, is the normalized time step t The second fusion feature of and The time steps The mean and standard deviation of the fusion features, and is a learnable parameter; The second fusion feature is compressed through a preset fully connected layer to generate a third fusion feature; The output layer outputs the third fused feature sequence of all time steps as the fused feature.

[0011] Furthermore, the dual-branch architecture includes a first branch and a second branch; The first branch is a classification branch, which is composed of a fully connected layer and a Softmax layer and is used to generate a probability distribution of the character category; The second branch is a regression branch, which is composed of a fully connected layer and a reshaping layer, and is used to generate a spatiotemporal coordinate sequence of the writing trajectory; The backbone of the multi-task collaborative model is composed of a convolutional neural network and a long short-term memory network with residual connections.

[0012] Furthermore, the first branch converts the cross entropy loss As the loss function, the difference between the probability distribution of the predicted character category and the actual character is calculated. The formula is as follows:

[0013] in, is the total number of character categories, For the m character categories one-hot encoding, For the m The predicted probability of each character category; The second branch uses regression loss as a loss function, where the regression loss is composed of a distance-based trajectory error loss and a similarity-based shape constraint loss; Trajectory error loss uses mean square error Measure the predicted spatiotemporal coordinate sequence and real space-time coordinate sequence B The point-to-point Euclidean distance is as follows:

[0014] in, is the total number of trajectory points in the space-time coordinate sequence, and are the real coordinates and predicted coordinates of the trajectory points in the spatiotemporal coordinate sequence respectively; Shape Constraint Loss The negative Pearson correlation coefficient is used to emphasize the overall similarity of trajectory shapes, and the formula is:

[0015] in, is the Pearson correlation coefficient, To predict the spatiotemporal coordinate sequence, B is the real space-time coordinate sequence; The trajectory error loss and shape constraint loss are weighted together to generate the regression loss :

[0016] in, and are distance loss weight and shape loss weight respectively; The classification loss and regression loss are weighted together to generate the multi-task total loss :

[0017] in, and are classification weight and regression weight respectively.

[0018] Furthermore, the dynamic adjustment strategy of the classification weight and regression weight is as follows: The ratio of the convergence speed is calculated based on the rate of change of the classification loss and the regression loss r ; when When , the value of regression weight is increased, otherwise the value of classification weight is increased; The dynamic weight averaging (DWA) mechanism is used to dynamically adjust the classification weights and regression weights.

[0019] Furthermore, the two training phases include a first training phase and a second training phase; The dynamic division mechanism of the two training stages is as follows: The first training stage improves the overall smoothness and shape consistency of the writing trajectory by minimizing the shape constraint loss, and increases the value of the shape loss weight in the regression loss to amplify the influence of the shape constraint; When the first average change rate of the shape similarity loss is less than a preset first threshold, it is determined that the shape similarity has initially converged, and the training phase is switched to the second training phase; The second training stage optimizes the absolute position accuracy of the trajectory points in the spatiotemporal coordinate sequence by increasing the distance loss weight in the regression loss, and stops model training when the second average change rate is less than a preset second threshold or reaches a preset maximum number of training times.

[0020] Furthermore, the first branch generates a probability distribution of character categories, specifically in the following steps: The fusion feature is transformed nonlinearly through the fully connected layer to generate a score vector of the character category ; The score vector Converted to the probability distribution of character categories through the Softmax layer:

[0021] in, For the m The probability of character categories, For the m The scores of character categories, is the total number of character categories; Select the character with the highest probability as the output result; The second branch generates a time-space coordinate sequence, and the specific steps are as follows: Expand the fusion features into multi-dimensional features through the fully connected layer : The reshape layer reshapes the multidimensional features into a sequence of spatiotemporal coordinates :

[0022] in, () is a dimension transformation operation.

[0023] Furthermore, the inertial measurement data is collected using a sensor-enhanced pen based on a multi-axis sensor, including an accelerometer, a gyroscope, a magnetometer, and a force sensor; The collection of the visual trajectory data is specifically as follows: 1) Video stream data of the writing process through the camera from outside to inside; 2) Compensating for blur defects in the image in the video stream data caused by rapid writing based on a preset event camera, specifically by the following operations: Asynchronously capturing pixel-level brightness changes of an image in the video stream data by the event camera to generate event stream data, and fusing the event stream data with the video stream data by using a spatiotemporal registration formula to generate a fused image; 3) Process the fused image in the compensated video stream data based on the preset U-Net segmentation network and output the probability distribution of the pen tip coordinates; 4) Take the top 20 pen tip coordinates with the highest probability and calculate the median value as the pen tip coordinate to generate visual trajectory data.

[0024] Compared with the prior art, the advantages of the present invention are: 1. This invention effectively integrates complementary information by synchronously collecting inertial measurement and visual trajectory data, combining dynamic alignment and cross-modal attention mechanisms, significantly improving robustness to writing jitter, occlusion, and noise, and overcoming the inherent defects of a single modality. 2. This paper constructs a dual-branch model to simultaneously optimize trajectory generation and character recognition, and uses a multi-objective loss function to achieve joint optimization of trajectory local accuracy and global structural similarity, avoiding the error accumulation problem of traditional separate tasks; 3. The present invention adopts a two-stage training strategy with dynamic division. The first training stage strengthens the smoothness of the overall trajectory shape, and the second training stage focuses on the accuracy of point-level coordinates. Combined with the dynamic weight adjustment mechanism, it adaptively balances the convergence speed of different tasks, thereby improving the model training efficiency and generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] Figure 1 Schematic diagram of the system workflow of the present invention; Figure 2 A schematic diagram of the fusion feature generation process of the present invention; Figure 3 Schematic diagram of the multi-task collaborative model of the present invention. DETAILED DESCRIPTION

[0027] To achieve the above objectives, the present invention is implemented through the following technical solutions: the present invention provides a writing trajectory optimization system based on machine learning, please refer to Figure 1 , the system comprises: Data acquisition module: collects first writing data of the writing process based on a preset synchronization strategy, including inertial measurement data and visual trajectory data; preprocesses the first writing data to generate second writing data, wherein the preprocessing includes dynamic length alignment, coordinate system one, and dynamic visual compensation; In this embodiment, the first writing data is the original multivariate time series data when writing on the electronic whiteboard. The collection process is as follows: Inertial measurement data is collected using the sensor-enhanced pen, including multi-axis sensor data: Accelerometer: captures the linear acceleration of the writing instrument (3D data); Gyroscope: measures changes in angular velocity (3D data); Magnetometer: detects direction information (3D data); Force sensor: records pen tip pressure (1D data); The collection of visual trajectory data is as follows: 1) Using an outside-in camera (such as a Kinect system or a custom camera array) to capture the video stream data of the writing process; 2) Based on the preset event camera, the image blur defects in the video stream data caused by fast writing are compensated. The specific operations are as follows: The event camera asynchronously captures the pixel-level brightness changes of the image in the video stream data to generate event stream data. The event stream data is then fused with the video stream data using the spatiotemporal registration formula to compensate for the blur defects in the video stream data. The formula is as follows:

[0028] in, is the fused image after compensation, is the image in the video stream, is the dynamic fusion weight (based on the event density function), is the preset time window (covering half of the writing jitter cycle), is the absolute value of the brightness change, The preset trigger threshold (dynamically adjusted based on light intensity) preserves the high-frequency pen tip motion details after fusion, eliminating trajectory breaks caused by motion blur. is the event activation function, The value is 1 when it is set, otherwise it is 0. For events a The event intensity (the value generated by the event camera), For events to occur time point; 3) Based on the preset U-Net segmentation network (parameters ) Process the video frames in the compensated video stream data , output the probability distribution of the pen tip coordinates:

[0029] in, To represent the pixel position (i,j) The conditional probability that there is a pen tip at , Timestamp t The fused image at is the mapping function of the U-Net segmentation network, 4) Take the top 20 pen tip coordinates with the highest probability and calculate the median value as the pen tip coordinate to generate visual trajectory data.

[0030] Synchronization strategies include hardware-level synchronization and software-level synchronization; Hardware-level synchronization: The sensor enhancement pen and camera use a unified clock source (e.g., an onboard timer) to ensure that the timestamps of each data point are aligned; Software-level synchronization: The timestamp matching algorithm in the acquisition software implements software-level synchronization, for example, by compensating for small delays based on interpolation:

[0031] in, is the preset reference timestamp, timestamps for sensor-enhanced pens, is the timestamp of the camera, and are the offsets of the sensor enhancement pen and camera after calibration, respectively.

[0032] The second writing data is standardized time series data generated after preprocessing the first writing data; Dynamic length alignment takes the character with the longest collection time in the first writing data as the benchmark and sets the time threshold ,For the characters whose collection duration is less than the time threshold, zero ,filling is performed at the end of their time series data; Coordinate system normalization converts the pen tip coordinates to the same millimeter-level physical coordinate system as the inertial data:

[0033] in, and is the converted pen tip coordinate, and is the pen tip coordinate, is the preset calibration matrix, obtained through checkerboard calibration, It is a preset offset vector to compensate for equipment installation errors.

[0034] Feature extraction and fusion module: extracts and fuses features of the second written data using a preset feature model to generate fusion features; the feature model is constructed based on a convolutional neural network and a long short-term memory network; The specific steps for generating fusion features are as follows. Figure 2 : The feature model includes input layer, feature extraction layer, time series modeling layer, feature fusion layer, feature enhancement layer and output layer; The second writing data is divided into multiple local segments using a sliding window, where the window size is set to 5 sampling points (covering about 50ms of writing dynamics); The input layer inputs local fragments; The feature extraction layer is composed of a convolutional neural network, including temporal convolution and feature dimensionality reduction; Temporal convolution uses two layers of 1D-CNN to capture the local spatiotemporal patterns of local segments to generate abstract features: The first layer consists of 64 pixels of size Convolution kernel, output basic features ,in is the maximum time step:

[0035] in, is the time step t The basic characteristics of activation function, is the time step To time step The input fragment when is a one-dimensional convolution operation, and is a learnable parameter; The second layer consists of 128 pixels of size Convolution kernel, output abstract features ;

[0036] in, is the time step t The abstract characteristics of is the time step To time step The basic characteristics of and is a learnable parameter; Feature dimensionality reduction compresses the abstract features through the largest pooling layer (pooling kernel size = 2) to generate compressed features , retaining significant local features; The time series modeling layer consists of a two-layer bidirectional long short-term memory network (LSTM); Suppress features Enter the bidirectional LSTM: The forward layer and the backward layer each contain 128 hidden units, which process the compressed features in the forward and reverse order of time respectively. ; Generate bidirectional hidden state and , and the bidirectional hidden state is concatenated into the final hidden state :

[0037] The first layer of LSTM is based on the final hidden state Generate the first time series feature ; The second layer is based on the first temporal features Generate the second time series feature ; The feature fusion layer compresses features and the second timing characteristics To perform cross-modal feature fusion, the specific steps are as follows: 1) Compress features and Second timing characteristics Align according to the preset timeline; 2) Dynamically fuse compressed features through learnable weight matrices and the second timing characteristics Generate the first fusion feature:

[0038] in, is the time step t The first fusion feature when is the time step t The fusion weight when is element-wise multiplication; The fusion weight is obtained through the attention weight formula, which is as follows:

[0039] in, and is a learnable parameter, is the activation function; The feature enhancement layer performs layer normalization on the fused features to improve training stability:

[0040] in, is the normalized time step t The second fusion feature of and is the time step The mean and standard deviation of the fusion features, and is a learnable parameter; The normalized second fusion feature is compressed to 256 dimensions through the preset fully connected layer to generate the third fusion feature ; The output layer takes the third fused feature sequence of all time steps as the fused feature, and the fused feature sequence is as follows:

[0041] in, K is the number of time steps.

[0042] Model construction module: Constructing a multi-task collaborative model, the multi-task collaborative model adopts a dual-branch architecture, and setting different loss functions based on the dual-branch architecture; training the multi-task collaborative model in two training phases based on dynamic partitioning, including optimizing the shape similarity of writing trajectories in the first training phase and enhancing the distance accuracy of writing trajectory points in the second training phase; The specific structure of the dual-branch architecture is as follows, please refer to Figure 3 : The first branch is the classification branch, which consists of a fully connected layer and a Softmax layer and is used to generate the probability distribution of character categories; The second branch is the regression branch, which consists of a fully connected layer and a reshaping layer, and is used to generate the spatiotemporal coordinate sequence of the writing trajectory.

[0043] The backbone of the multi-task collaborative model is composed of a convolutional neural network (CNN) and a long short-term memory network (LSTM) with residual connections; The loss function of the multi-task collaborative model is designed as follows: The first branch uses the cross entropy loss As the loss function, the difference between the probability distribution of the predicted character category and the actual character is calculated. The formula is as follows:

[0044] in, is the total number of character categories, For the m One-hot encoding of character categories (when the predicted character category is consistent with the true character category, the value is 1, otherwise it is 0), For the m The predicted probability of the character category.

[0045] The second branch uses regression loss as the loss function, which is composed of trajectory error loss based on distance and shape constraint loss based on similarity; Trajectory error loss uses mean square error Measure the predicted spatiotemporal coordinate sequence and real space-time coordinate sequence B The point-to-point Euclidean distance is as follows:

[0046] in, is the total number of trajectory points in the space-time coordinate sequence, and are the real coordinates and predicted coordinates of the trajectory points in the spatiotemporal coordinate sequence respectively; Shape Constraint Loss The negative Pearson correlation coefficient is used to emphasize the overall similarity of trajectory shapes (invariant to scaling and translation), and the formula is:

[0047] in, is the Pearson correlation coefficient, To predict the spatiotemporal coordinate sequence, B is the real space-time coordinate sequence; The calculation formula of Pearson correlation coefficient is as follows:

[0048] in, and is the mean of the true coordinates of the trajectory points in the space-time coordinate sequence, and is the mean of the predicted coordinates of the trajectory points in the spatiotemporal coordinate sequence. This loss ensures that the shape of the trajectory changes consistently along the time axis. The trajectory error loss and shape constraint loss are weighted and generated into regression loss :

[0049] in, and are the distance loss weight and shape loss weight, which are adjustable weight coefficients and their initial values ​​are both set to 0.5 (the two loss terms are treated equally in the early stage of training to avoid one task dominating the optimization process); The classification loss and regression loss are weighted together to generate the multi-task total loss :

[0050] in, and are classification weight and regression weight respectively; The dynamic adjustment strategy of classification weight and regression weight is as follows: Calculate the convergence speed based on the rate of change of classification loss and regression loss , the formula is as follows:

[0051] in, and are the changes in classification loss and regression loss in adjacent training rounds, is the time interval between adjacent training rounds; A training round refers to the time it takes for the multi-task collaborative model to fully traverse the entire training dataset during training. when When , the value of regression weight is increased, otherwise the value of classification weight is increased; The dynamic weight averaging (DWA) mechanism is used to dynamically adjust the classification weight and regression weight. The formula is as follows:

[0052] in, and The time steps The classification weight and regression weight when is the preset learning rate (the default value is 0.01. The default learning rate of 0.01 is intended to ensure the dynamic balance of the weights of classification and regression tasks through gradual adjustment with small steps, avoid training oscillations, and match the gradient magnitude difference with the stability requirements of the two-stage training strategy). and The time steps The classification weight and regression weight when () is an exponential function, and They are the partial derivatives of the total loss with respect to the classification weight and the partial derivative of the total loss with respect to the regression weight respectively; Normalize the classification weights and regression weights to ensure For example, if the regression loss variance is too high (indicating instability), increase the shape constraint weight To smooth learning; During the training process, the weights are updated at fixed intervals, giving priority to tasks that converge slowly. For example, when the distance loss in the regression task fluctuates greatly, the system automatically increases the weights. And fine-tune the ratio of distance loss weight and shape loss weight to ensure the co-evolution of classification and regression tasks to avoid dominance of one side.

[0053] The dynamic division mechanism of the two training stages is as follows: Phase switching is based on real-time monitoring of the task's convergence status (such as loss change rate and stability indicators) to ensure logical rigor and adaptability: The first training phase minimizes the shape constraint loss To improve the overall smoothness and shape consistency of the writing trajectory (reduce the variance and jitter of the writing trajectory), and increase the value of the shape loss weight in the regression loss to amplify the influence of the shape constraint; When the first average change rate of shape similarity loss Less than the preset first threshold (the range is [0.003, 0.008], when When the multi-task collaborative model is in the complete convergence period (the actual benefit of continuing to optimize the shape similarity loss is the smallest), the shape similarity is judged to have initially converged and the training phase is switched to the second phase; The second training phase optimizes the absolute position accuracy of trajectory points in the spatiotemporal coordinate sequence by increasing the distance loss weight in the regression loss and monitoring the second average change rate of the distance loss. , when the second average rate of change Less than the preset second threshold (the range is [5×10 -7 ,5×10 -6 ], based on the physical accuracy requirements of the pen tip coordinates) or when the preset maximum number of training times is reached, the model training is stopped; Trajectory generation and recognition module: inputs the fusion features into the multi-task collaborative model to generate the probability distribution of character categories and the spatiotemporal coordinate sequence of the writing trajectory; The first branch generates the probability distribution of character categories. The specific steps are as follows: The fusion features are transformed nonlinearly through the fully connected layer to generate the score vector of the character category. :

[0054] in, To fusion features, and are learnable parameters.

[0055] Will Converted to the probability distribution of character categories through the Softmax layer:

[0056] in, For the m The probability of character categories, For the m The scores of character classes, is the total number of character categories; Select the character with the highest probability as the output result; The second branch generates a sequence of space-time coordinates. The specific steps are as follows: Expand the fused features into multi-dimensional features through the fully connected layer:

[0057] The reshape layer reshapes the multidimensional features into a sequence of spatiotemporal coordinates :

[0058] in, () It is a dimension transformation operation.

[0059] In summary, the present invention constructs a dual-branch multi-task collaborative model, in which the classification branch focuses on character category prediction, and the regression branch is responsible for generating spatiotemporal coordinate sequences. It also designs a multi-objective optimization function that integrates cross-entropy loss, trajectory distance error, and shape constraint loss, thereby achieving end-to-end joint optimization of trajectory reconstruction and character recognition tasks, effectively avoiding the error accumulation problem caused by task splitting.

[0060] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A writing trajectory optimization system based on machine learning, characterized in that: include: Data acquisition module: collects first writing data of the writing process based on a preset synchronization strategy, including inertial measurement data and visual trajectory data; Preprocessing the first writing data to generate second writing data, wherein the preprocessing includes dynamic length alignment, coordinate system one, and dynamic visual compensation; Feature extraction and fusion module: extracts and fuses features of the second written data using a preset feature model to generate fusion features; the feature model is constructed based on a convolutional neural network and a long short-term memory network; Model construction module: Constructing a multi-task collaborative model, the multi-task collaborative model adopts a dual-branch architecture, and setting different loss functions based on the dual-branch architecture; training the multi-task collaborative model in two training phases based on dynamic partitioning, including optimizing the shape similarity of writing trajectories in the first training phase and enhancing the distance accuracy of writing trajectory points in the second training phase; Trajectory generation and recognition module: inputs the fusion features into the multi-task collaborative model to generate the probability distribution of character categories and the spatiotemporal coordinate sequence of the writing trajectory.

2. The system according to claim 1, wherein: The feature model includes an input layer, a feature extraction layer, a time series modeling layer, a feature fusion layer, a feature enhancement layer and an output layer; dividing the second written data into a plurality of local segments through a preset sliding window; The input layer inputs the local segment; The feature extraction layer is composed of a convolutional neural network, including temporal convolution and feature dimensionality reduction; The temporal convolution uses two layers of 1D-CNN to capture the local spatiotemporal patterns of local segments to generate abstract features; The feature dimensionality reduction compresses the abstract features through the pooling layer to generate compressed features ; The temporal modeling layer is composed of a two-layer bidirectional long short-term memory network, based on the compression feature Generate the second time series feature .

3. The system according to claim 2, characterized in that The feature fusion layer combines the compressed features And the second timing characteristics Perform cross-modal feature fusion to generate the fusion feature. The specific steps are as follows: 1) The compression feature And the second timing characteristics Align according to the preset timeline; 2) Fusion of the compression features And the second timing characteristics Generate the first fusion feature: in, is the time step t The first fusion feature when is the time step t The fusion weight when is element-wise multiplication; The fusion weight is obtained through the attention weight formula, which is as follows: in, and is a learnable parameter, is the activation function; The feature enhancement layer normalizes the first fusion feature: in, is the normalized time step t The second fusion feature of and The time steps The mean and standard deviation of the fusion features, and is a learnable parameter; The second fusion feature is compressed through a preset fully connected layer to generate a third fusion feature; The output layer outputs the third fused feature sequence of all time steps as the fused feature.

4. The system according to claim 1, wherein: The dual-branch architecture includes a first branch and a second branch; The first branch is a classification branch, which is composed of a fully connected layer and a Softmax layer and is used to generate a probability distribution of the character category; The second branch is a regression branch, which is composed of a fully connected layer and a reshaping layer, and is used to generate a spatiotemporal coordinate sequence of the writing trajectory; The backbone of the multi-task collaborative model is composed of a convolutional neural network and a long short-term memory network with residual connections.

5. The system according to claim 4, characterized in that The first branch uses the cross entropy loss As the loss function, the difference between the probability distribution of the predicted character category and the actual character is calculated. The formula is as follows: in, is the total number of character categories, For the m character categories one-hot encoding, For the m The predicted probability of each character category; The second branch uses regression loss as a loss function, where the regression loss is composed of a distance-based trajectory error loss and a similarity-based shape constraint loss; Trajectory error loss uses mean square error Measure the predicted spatiotemporal coordinate sequence and real space-time coordinate sequence B The point-to-point Euclidean distance is as follows: in, is the total number of trajectory points in the space-time coordinate sequence, and are the real coordinates and predicted coordinates of the trajectory points in the spatiotemporal coordinate sequence respectively; Shape Constraint Loss The negative Pearson correlation coefficient is used to emphasize the overall similarity of trajectory shapes, and the formula is: in, is the Pearson correlation coefficient, To predict the spatiotemporal coordinate sequence, B is the real space-time coordinate sequence; The trajectory error loss and shape constraint loss are weighted together to generate the regression loss : in, and are distance loss weight and shape loss weight respectively; The classification loss and regression loss are weighted together to generate the multi-task total loss : in, and are classification weight and regression weight respectively.

6. The system according to claim 5, characterized in that The dynamic adjustment strategy of the classification weight and regression weight is as follows: The ratio of the convergence speed is calculated based on the rate of change of the classification loss and the regression loss r ; when When , the value of regression weight is increased, otherwise the value of classification weight is increased; The dynamic weight averaging (DWA) mechanism is used to dynamically adjust the classification weights and regression weights.

7. The system according to claim 6, characterized in that The two training stages include a first training stage and a second training stage; The dynamic division mechanism of the two training stages is as follows: The first training stage improves the overall smoothness and shape consistency of the writing trajectory by minimizing the shape constraint loss, and increases the value of the shape loss weight in the regression loss to amplify the influence of the shape constraint; When the first average change rate of the shape similarity loss is less than a preset first threshold, it is determined that the shape similarity has initially converged, and the training phase is switched to the second training phase; The second training stage optimizes the absolute position accuracy of the trajectory points in the spatiotemporal coordinate sequence by increasing the distance loss weight in the regression loss, and stops model training when the second average change rate is less than a preset second threshold or reaches a preset maximum number of training times.

8. The system according to claim 7, characterized in that The first branch generates a probability distribution of character categories, specifically in the following steps: The fusion feature is transformed nonlinearly through the fully connected layer to generate a score vector of the character category ; The score vector Converted to the probability distribution of character categories through the Softmax layer: in, For the m The probability of character categories, For the m The scores of character categories, is the total number of character categories; Select the character with the highest probability as the output result; The second branch generates a time-space coordinate sequence, and the specific steps are as follows: Expand the fusion features into multi-dimensional features through the fully connected layer : The reshape layer reshapes the multidimensional features into a sequence of spatiotemporal coordinates : in, () It is a dimension transformation operation.

9. The system according to claim 1, wherein: The inertial measurement data is collected using a sensor-enhanced pen based on multi-axis sensors, including accelerometers, gyroscopes, magnetometers, and force sensors; The collection of the visual trajectory data is specifically as follows: 1) Video stream data of the writing process through the camera from outside to inside; 2) Compensating for blur defects in the image in the video stream data caused by rapid writing based on a preset event camera, specifically by the following operations: Asynchronously capturing pixel-level brightness changes of an image in the video stream data by the event camera to generate event stream data, and fusing the event stream data with the video stream data by using a spatiotemporal registration formula to generate a fused image; 3) Process the fused image in the compensated video stream data based on the preset U-Net segmentation network and output the probability distribution of the pen tip coordinates; 4) Take the top 20 pen tip coordinates with the highest probability and calculate the median value as the pen tip coordinate to generate visual trajectory data.

Images