Machine learning based writing trajectory optimization system

Abstract

The application provides a handwriting trajectory optimization system based on machine learning, relates to the technical field of machine learning and artificial intelligence, and comprises a data acquisition module, which synchronously acquires inertial measurement and visual trajectory data, generates standardized handwriting data through dynamic alignment and coordinate unification; a feature extraction and fusion module, which extracts space-time features and fuses multi-source features through a cross-modal attention mechanism; a model construction module, which constructs a double-branch multi-task collaborative model, a classification branch predicts character categories, a regression branch generates space-time coordinate sequences, and a multi-objective optimization function is designed by fusing cross-entropy loss, trajectory distance error and shape constraint loss; and a trajectory generation and recognition module, which is based on a two-stage training strategy of dynamic division, the first training stage optimizes the overall shape similarity of the trajectory, and the second training stage strengthens the distance accuracy of the trajectory points, so that high-precision trajectory reconstruction and character recognition are realized. The application effectively solves the handwriting jitter problem and improves the trajectory smoothness and recognition robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning and artificial intelligence, in particular to a writing trajectory optimization system based on machine learning. BACKGROUND

[0002] In interactive writing scenarios such as electronic whiteboards, user writing trajectories are easily disturbed by factors such as hand jitter, sensor noise, and visual occlusion, resulting in significant jitter and distortion in the collected trajectory data. Traditional trajectory capture systems based on a single modality (such as pure vision or pure inertial sensing) have inherent limitations: visual trajectories are easily affected by changes in viewing angle, lighting conditions, and occlusion, especially in multi-user collaborative electronic whiteboard scenarios, where hand occlusion often causes pen tip positioning to drift; while inertial measurement units (IMUs) can capture subtle movements, they have cumulative errors and lack absolute position. Although multi-modal data fusion has been proposed to complement the advantages, existing methods still have deficiencies in terms of timing alignment accuracy and cross-modal feature collaboration, making it difficult to balance trajectory smoothness and recognition robustness.

[0003] Existing trajectory optimization schemes usually use post-processing filtering (such as Kalman filtering) or independent training recognition models, which have obvious limitations: filtering methods can suppress high-frequency noise, but may over-smooth writing details, resulting in loss of stroke transition features; while treating trajectory generation and character recognition as separate tasks, the inherent correlation between the two is ignored, which can easily cause error accumulation, in addition, traditional models are difficult to optimize both local point accuracy (such as pen tip coordinate error) and global shape similarity (such as stroke structure fidelity), and in real-time writing scenarios on electronic whiteboards, it is often difficult to balance jitter suppression and writing intent preservation. Therefore, there is an urgent need for an end-to-end learning framework that can simultaneously optimize trajectory reconstruction and character recognition to improve the accuracy and smoothness of the electronic whiteboard writing experience. SUMMARY

[0004] To solve the technical problems mentioned in the background art, the purpose of the present application is to provide a writing trajectory optimization system based on machine learning.

[0005] In order to achieve the above-mentioned purpose, the technical scheme adopted by the present application is as follows:

[0006] The writing trajectory optimization system based on machine learning comprises:

[0007] The data acquisition module acquires first writing data based on a predetermined synchronization strategy, including inertial measurement data and visual trajectory data; the first writing data is preprocessed to generate second writing data, the preprocessing including dynamic length alignment, coordinate system unification, and dynamic visual compensation;

[0008] The feature extraction and fusion module: a preset feature model is used to perform feature extraction and fusion on the second writing data to generate fused features; the feature model is constructed based on a convolutional neural network and a long short-term memory network;

[0009] The model construction module: a multi-task collaborative model is constructed, the multi-task collaborative model is a double-branch architecture, different loss functions are set based on the double-branch architecture; the multi-task collaborative model is trained based on two training stages divided dynamically, including a first training stage for optimizing shape similarity of writing trajectories and a second training stage for strengthening distance accuracy of writing trajectory points;

[0010] The double-branch architecture includes a first branch and a second branch;

[0011] The first branch is a classification branch, which is composed of a full connection layer and a Softmax layer, and is used to generate a probability distribution of the character category;

[0012] The second branch is a regression branch, which is composed of a full connection layer and a reshaping layer, and is used to generate a spatiotemporal coordinate sequence of the writing trajectory;

[0013] The backbone of the multi-task collaborative model is composed of a convolutional neural network and a long short-term memory network in residual connection;

[0014] The two training stages include a first training stage and a second training stage;

[0015] The dynamic division mechanism of the two training stages is as follows:

[0016] The first training stage improves the overall smoothness and shape consistency of the writing trajectory by minimizing the shape constraint loss, and increases the value of the shape loss weight in the regression loss to amplify the influence of the shape constraint;

[0017] When the first average change rate of the shape similarity loss is less than a preset first threshold, it is determined that the shape similarity is preliminarily converged, and the second training stage is switched to;

[0018] The second training stage optimizes the absolute position accuracy of the trajectory points in the spatiotemporal coordinate sequence by increasing the distance loss weight in the regression loss, and stops model training when the second average change rate is less than a preset second threshold or reaches a preset maximum training number;

[0019] The trajectory generation and recognition module: the fused features are input into the multi-task collaborative model to generate a probability distribution of the character category and a spatiotemporal coordinate sequence of the writing trajectory.

[0020] Further, the feature model includes an input layer, a feature extraction layer, a time series modeling layer, a feature fusion layer, a feature enhancement layer, and an output layer;

[0021] The second writing data is divided into multiple local segments by a preset sliding window;

[0022] The input layer inputs the local segments;

[0023] The feature extraction layer is composed of a convolutional neural network, including a time sequence convolution and feature dimension reduction;

[0024] The time sequence convolution adopts two layers of 1D-CNN to capture the local space-time pattern of the local segment to generate abstract features;

[0025] The feature dimension reduction compresses the abstract features through a pooling layer to generate compressed features ;

[0026] The time sequence modeling layer is composed of a double-layer bidirectional long short-term memory network, and generates second time sequence features based on the compressed features .

[0027] Further, the feature fusion layer performs cross-modal feature fusion on the compressed features and the second time sequence features to generate the fusion features, and the specific steps are as follows:

[0028] 1) The compressed features and the second time sequence features are aligned according to a preset time axis;

[0029] 2) The compressed features and the second time sequence features are fused to generate first fusion features:

[0030]

[0031] wherein, is the first fusion feature at time step t , is the fusion weight at time step t , is an element-wise multiplication, is the compressed feature at time step t , is the second time sequence feature at time step t ;

[0032] The fusion weight is obtained through an attention weight formula, and the formula is as follows:

[0033]

[0034] wherein, and For learnable parameters, For activation function, For splicing and Operation;

[0035] The feature enhancement layer normalizes the first fused feature:

[0036]

[0037] in, For normalized time steps t The second fusion feature of time, and They are time steps Mean and standard deviation of time-fusion features and These are learnable parameters;

[0038] The second fusion feature is compressed using a pre-defined fully connected layer to generate the third fusion feature;

[0039] The output layer outputs the third fusion feature sequence of all time steps as the fusion feature.

[0040] Furthermore, the first branch will have cross-entropy loss. As the loss function, the difference between the probability distribution of the predicted character category and the actual character is calculated, as shown in the following formula:

[0041]

[0042] in, The total number of character categories, For the first m One-hot encoding for each character category. For the first m The predicted probability of each character category;

[0043] The second branch uses regression loss as the loss function, which consists of distance-based trajectory error loss and similarity-based shape constraint loss;

[0044] Trajectory error loss uses mean square error Measuring the predicted spatiotemporal coordinate sequence With the real spatiotemporal coordinate sequence B The point-to-point Euclidean distance is calculated using the following formula:

[0045]

[0046] in, The total number of trajectory points in the spatiotemporal coordinate sequence. and are the real and predicted coordinates of the trajectory point in the sequence of spatiotemporal coordinates, respectively;

[0047] shape constraint loss The negative Pearson correlation coefficient is used to emphasize the overall similarity of the trajectory shape, and the formula is:

[0048]

[0049] wherein, is the Pearson correlation coefficient, is the predicted sequence of spatiotemporal coordinates, B is the real sequence of spatiotemporal coordinates;

[0050] The regression loss is generated by weighted combination of the trajectory error loss and the shape constraint loss :

[0051]

[0052] wherein, and are the distance loss weight and the shape loss weight, respectively;

[0053] The multi-task total loss is generated by weighted combination of the classification loss and the regression loss :

[0054]

[0055] wherein, and are the classification weight and the regression weight, respectively.

[0056] Further, the dynamic adjustment strategy of the classification weight and the regression weight is as follows:

[0057] The ratio of the convergence speed is calculated based on the change rate of the classification loss and the regression loss r ;

[0058] When the value of the regression weight is increased, otherwise the value of the classification weight is increased;

[0059] The dynamic weight average (DWA) mechanism is used to dynamically adjust the classification weight and the regression weight.

[0060] Further, the first branch generates the probability distribution of the character category, and the specific steps are as follows:

[0061] The fusion feature is nonlinearly transformed by the fully connected layer to generate a score vector of the character category ;

[0062] The score vector is The probability distribution of character categories converted through the Softmax layer is as follows:

[0063]

[0064] in, For the first m The probability of each character category. For the first m The score for each character category, Total number of character categories;

[0065] Select the character with the highest probability as the output result;

[0066] The second branch generates a spatiotemporal coordinate sequence, and the specific steps are as follows:

[0067] The fused features are extended into multidimensional features through the fully connected layer. :

[0068] The reshaping layer reshapes the multidimensional features into a predicted spatiotemporal coordinate sequence. :

[0069]

[0070] in, () represents a dimension transformation operation.

[0071] Furthermore, the inertial measurement data is acquired using a sensor-enhanced pen based on multi-axis sensors, including accelerometers, gyroscopes, magnetometers, and force sensors;

[0072] The visual trajectory data was collected in the following specific ways:

[0073] 1) Video stream data of the writing process via the camera from the outside in;

[0074] 2) Based on a preset event camera, compensation is provided for blurring defects in the video stream data caused by rapid image processing. The specific operation is as follows:

[0075] The event camera asynchronously captures pixel-level brightness changes in the video stream data to generate event stream data, and then fuses the event stream data with the video stream data using a spatiotemporal registration formula to generate a fused image.

[0076] 3) Based on the preset U-Net segmentation network, process the fused image in the compensated video stream data and output the probability distribution of pen tip coordinates;

[0077] 4) Take the pen tip coordinates of the top 20 with the highest probability, calculate the median as the pen tip coordinate to generate visual trajectory data.

[0078] Compared with the prior art, the application has the advantages that:

[0079] 1、The application effectively fuses complementary information by synchronously collecting inertial measurement and visual trajectory data, combining dynamic alignment and cross-modal attention mechanism, significantly improves the robustness to writing jitter, occlusion and noise, and overcomes the inherent defects of single modal;

[0080] 2、The application synchronously optimizes trajectory generation and character recognition by constructing a double-branch model, and uses a multi-objective loss function to realize the joint optimization of local precision and global structural similarity of the trajectory, avoiding the error accumulation problem of traditional separate tasks;

[0081] 3、The application uses a two-stage training strategy, the first training stage strengthens the overall shape smoothness of the trajectory, the second training stage focuses on the point-level coordinate accuracy, and cooperates with a dynamic weight adjustment mechanism to adaptively balance the convergence speed of different tasks, improve the model training efficiency and generalization ability. BRIEF DESCRIPTION OF DRAWINGS

[0082] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0083] Figure 1 The system workflow schematic diagram of the present application;

[0084] Figure 2 The fusion feature generation process schematic diagram of the present application;

[0085] Figure 3 The multi-task collaborative model schematic diagram of the present application. DETAILED DESCRIPTION

[0086] To achieve the above purpose, the present application is realized by the following technical solutions, the present application provides a writing trajectory optimization system based on machine learning, please refer to Figure 1 , the system comprises:

[0087] Data acquisition module: based on the preset synchronization strategy, collecting the first writing data of the writing process, including inertial measurement data and visual trajectory data; the first writing data is preprocessed to generate the second writing data, the preprocessing includes dynamic length alignment, coordinate system and dynamic visual compensation;

[0088] In this embodiment, the first writing data is the original multi-element time series data when writing on the electronic whiteboard, and the collection process is as follows:

[0089] Inertial measurement data was acquired using a sensor augmentation pen, including multi-axis sensor data:

[0090] Accelerometer: Captures the linear acceleration (3D data) of writing instruments;

[0091] Gyroscope: measures changes in angular velocity (3D data);

[0092] Magnetometer: detects directional information (3D data);

[0093] Force sensor: records pen tip pressure (1D data);

[0094] The visual trajectory data collection process is as follows:

[0095] 1) The video stream data of the writing process is captured by an outside-in camera (such as the Kinect system or a custom camera array);

[0096] 2) Based on preset event camera compensation, the image blurring defect in the video stream data caused by rapid writing is compensated. The specific operation is as follows:

[0097] By asynchronously capturing pixel-level brightness changes in images from video stream data using an event camera, event stream data is generated. This event stream data is then fused with the video stream data using a spatiotemporal registration formula to compensate for blur defects in the video stream data. The formula is as follows:

[0098]

[0099] in, For the compensated fused image, For images in the video stream, For dynamic fusion weights (based on event density function). For a preset time window (covering half of the writing jitter cycle), This represents the absolute value of the change in brightness. Using a preset trigger threshold (dynamically adjusted based on light intensity), the fused data retains high-frequency pen tip motion details and eliminates trajectory breaks caused by motion blur. For event activation functions, The value is 1 if the condition is met, and 0 otherwise. For the event a The event intensity (the value generated by the event camera), For the occurrence of the event The point in time;

[0100] 3) Based on the preset U-Net segmentation network (parameters) Processing the video frames in the compensated video stream data , the probability distribution of the output pen tip coordinates:

[0101]

[0102] wherein, is the conditional probability of the presence of a pen tip at pixel position (i,j) is the fusion image at time stamp is the mapping function of the U-Net segmentation network, t is the mapping function of the U-Net segmentation network,

[0103] 4) Take the top 20 pen tip coordinates with the highest probability, and calculate the median as the pen tip coordinates to generate visual trajectory data.

[0104] The synchronization strategy includes hardware-level synchronization and software-level synchronization;

[0105] Hardware-level synchronization: the sensor-enhanced pen and the camera use a unified clock source (e.g., an on-board timer) to ensure that the timestamps of each data point are aligned;

[0106] Software-level synchronization: the timestamp matching algorithm in the acquisition software implements software-level synchronization, such as compensating for minor delays based on interpolation methods:

[0107]

[0108] wherein, is the preset reference timestamp, is the timestamp of the sensor-enhanced pen, is the timestamp of the camera, and are the offsets of the sensor-enhanced pen and the camera after calibration, respectively.

[0109] The second writing data is standardized time series data generated after preprocessing the first writing data;

[0110] Dynamic length alignment takes the character with the longest collection duration in the first writing data as the reference, and sets a time threshold For characters with a collection duration less than the time threshold, zero padding is performed at the end of their time series data.

[0111] Coordinate system normalization converts the pen tip coordinates to the same millimeter-level physical coordinate system as the inertial data:

[0112]

[0113] wherein, and are the converted pen tip coordinates, and are the pen tip coordinates, ​The calibration matrix is ​​obtained through checkerboard calibration. This is a preset offset vector used to compensate for equipment installation errors.

[0114] Feature extraction and fusion module: Extracts and fuses features from the second written data using a preset feature model to generate fused features; the feature model is constructed based on a convolutional neural network and a long short-term memory network;

[0115] The specific steps for generating fusion features are as follows; please refer to [link / reference]. Figure 2 :

[0116] The feature model includes an input layer, a feature extraction layer, a temporal modeling layer, a feature fusion layer, a feature enhancement layer, and an output layer;

[0117] The second writing data is divided into multiple local segments using a sliding window, with the window size set to 5 sampling points (covering approximately 50ms of writing dynamics).

[0118] The input layer inputs a local segment;

[0119] The feature extraction layer consists of a convolutional neural network, including temporal convolution and feature dimensionality reduction;

[0120] Temporal convolution uses two layers of 1D-CNN to capture the local spatiotemporal patterns of local segments and generate abstract features:

[0121] The first layer contains 64 items of various sizes. The convolution kernel outputs the basic features. ,in For the maximum time step:

[0122]

[0123] in, For time step t The basic characteristics of time, Activation function For time step Time step Input fragment at time, This is a one-dimensional convolution operation. and These are learnable parameters;

[0124] The second layer contains 128 items of different sizes. The convolution kernel outputs abstract features. ;

[0125]

[0126] in, For time step t Abstract characteristics of time, is the time step is the time step is the base feature at time step is the time step is the learnable parameter

[0127] The feature dimension reduction compresses the abstract feature to generate compressed feature through the max pooling layer (pooling kernel size = 2) , which retains the significant local feature

[0128] The time series modeling layer is composed of a double-layer bidirectional long short-term memory network (LSTM)

[0129] The compressed feature is input into the bidirectional LSTM

[0130] The forward layer and the backward layer each contain 128 hidden units, and process the compressed feature in time order and reverse order respectively ; generate bidirectional hidden state and , and the bidirectional hidden state is spliced into the final hidden state :

[0131]

[0132] The first layer LSTM generates the first time series feature based on the final hidden state ;

[0133] The second layer generates the second time series feature based on the first time series feature ;

[0134] The feature fusion layer performs cross-modal feature fusion on the compressed feature and the second time series feature , and the specific steps are as follows:

[0135] 1) Align the compressed feature and and the second time series feature according to the preset time axis

[0136] 2) Dynamically fuse the compressed feature and the second time series feature to generate the first fusion feature through the learnable weight matrix

[0137]

[0138] wherein, is the first fusion feature at time step t , and is the time step tTime-based fusion weights For element-wise multiplication, For time steps t Compression characteristics at time, For time steps t The second temporal characteristic of time;

[0139] The fusion weights are obtained through the attention weight formula, as follows:

[0140]

[0141] in, and For learnable parameters, For activation function, For splicing and Operation;

[0142] The feature enhancement layer performs layer normalization on the fused features to improve training stability.

[0143]

[0144] in, For normalized time steps t The second fusion feature of time, and For time steps Mean and standard deviation of time-fusion features and These are learnable parameters;

[0145] The normalized second fusion feature is compressed to 256 dimensions using a pre-defined fully connected layer to generate the third fusion feature. ;

[0146] The output layer uses the third fusion feature sequence of all time steps as the fusion feature, and the fusion feature sequence is as follows:

[0147]

[0148] in, K This represents the number of time steps.

[0149] Model building module: Constructs a multi-task collaborative model, which has a dual-branch architecture, and sets different loss functions based on the dual-branch architecture; trains the multi-task collaborative model based on two dynamically divided training phases, including a first training phase to optimize the shape similarity of the writing trajectory and a second training phase to enhance the distance accuracy of the writing trajectory points;

[0150] The double-branch architecture comprises a first branch and a second branch;

[0151] The first branch is a classification branch, which is composed of a full connection layer and a Softmax layer, and is used for generating a probability distribution of the character category;

[0152] The second branch is a regression branch, which is composed of a full connection layer and a reshaping layer, and is used for generating a sequence of spatiotemporal coordinates of the writing trajectory;

[0153] The backbone of the multi-task collaborative model is composed of a convolutional neural network and a long short-term memory network in a residual connection;

[0154] The two training stages comprise a first training stage and a second training stage;

[0155] The dynamic division mechanism of the two training stages is as follows:

[0156] The first training stage improves the overall smoothness and shape consistency of the writing trajectory by minimizing the shape constraint loss, and increases the value of the shape loss weight in the regression loss to amplify the influence of the shape constraint;

[0157] When the first average change rate of the shape similarity loss is less than a preset first threshold, it is determined that the shape similarity is preliminarily converged, and the second training stage is switched to;

[0158] The second training stage optimizes the absolute position accuracy of the trajectory points in the sequence of spatiotemporal coordinates by increasing the distance loss weight in the regression loss, and stops the model training when the second average change rate is less than a preset second threshold or reaches a preset maximum training number;

[0159] The specific structure of the double-branch architecture, please refer to Figure 3 :

[0160] The stage switching of the two training stages is based on real-time monitoring of the convergence state of the task (such as the loss change rate and stability index), ensuring logical rigor and self-adaptation:

[0161] The value range of the first threshold is [0.003, 0.008], when The multi-task collaborative model is in a complete convergence period, at which time the actual benefit of continuing to optimize the shape similarity loss is minimum;

[0162] The value range of the second threshold is [5×10−7, 5×10−6], based on the physical accuracy requirement of the pen tip coordinates;

[0163] The loss function design of the multi-task collaborative model is as follows:

[0164] The first branch will cross-entropy loss As the loss function, the difference between the probability distribution of the predicted character class and the real character is calculated, and the formula is as follows:

[0165]

[0166] wherein, is the total number of character classes, is the one-hot encoding of the m th character class (1 when the predicted character class is consistent with the real character class, otherwise 0), is the predicted probability of the m th character class.

[0167] The second branch takes the regression loss as the loss function, and the regression loss is composed of distance-based trajectory error loss and similarity-based shape constraint loss;

[0168] The trajectory error loss adopts the mean square error to measure the point-to-point Euclidean distance between the predicted spatiotemporal coordinate sequence and the real spatiotemporal coordinate sequence B , and the formula is as follows:

[0169]

[0170] wherein, is the total number of trajectory points in the spatiotemporal coordinate sequence, and are the real coordinates and predicted coordinates of the trajectory points in the spatiotemporal coordinate sequence, respectively;

[0171] The shape constraint loss adopts the negative Pearson correlation coefficient, which emphasizes the overall similarity of the trajectory shape (invariant to scaling and translation), and the formula is as follows:

[0172]

[0173] wherein, is the Pearson correlation coefficient, is the predicted spatiotemporal coordinate sequence, B is the real spatiotemporal coordinate sequence;

[0174] The calculation formula of the Pearson correlation coefficient is as follows:

[0175]

[0176] wherein, and are the mean values of the real coordinates of the trajectory points in the spatiotemporal coordinate sequence, and The loss ensures the consistency of the shape change of the trajectory in the time axis for the mean value of the predicted coordinates of the trajectory points in the sequence of space-time coordinates;

[0177] The trajectory error loss and the shape constraint loss are weighted and a regression loss is generated

[0178]

[0179] wherein, and are the distance loss weight and the shape loss weight respectively, and are adjustable weight coefficients, and the initial values are both set to 0.5 (to equally treat the two loss terms at the beginning of training, and to avoid that one task dominates the optimization process);

[0180] The classification loss and the regression loss are weighted and combined to generate a multi-task total loss

[0181]

[0182] wherein, and are the classification weight and the regression weight respectively;

[0183] The dynamic adjustment strategy of the classification weight and the regression weight is as follows:

[0184] The convergence speed is calculated based on the change rate of the classification loss and the regression loss , and the formula is as follows:

[0185]

[0186] wherein, and are the change amounts of the classification loss and the regression loss in adjacent training rounds respectively, is the time interval of adjacent training rounds;

[0187] A training round refers to that the multi-task collaborative model completely traverses the entire training data set once in the training process;

[0188] When , the value of the regression weight is increased, otherwise the value of the classification weight is increased;

[0189] The dynamic weight average (DWA) mechanism is adopted to dynamically adjust the classification weight and the regression weight, and the formula is as follows:

[0190]

[0191] wherein, and are the time steps ​​classification and regression weights at time step t, preset learning rate (default 0.01, the default learning rate 0.01 is designed to ensure the dynamic balance of classification and regression task weights through small stepwise adjustment, avoid training shock, and match the stability requirements of gradient magnitude difference and two-stage training strategy), and respectively, the classification and regression weights at time step t, classification and regression weights at time step t, () is an exponential function, and respectively, the partial derivative of the total loss with respect to the classification weight and the partial derivative of the total loss with respect to the regression weight;

[0192] The classification and regression weights are normalized to ensure For example, if the regression loss variance is too high (indicating instability), increase the shape constraint weight to smooth learning;

[0193] During training, weight updates are performed every fixed time, with priority response to slow-converging tasks, for example, when the distance loss in the regression task fluctuates greatly, the system automatically increases and fine-tunes the distance loss weight and shape loss weight ratio to ensure the co-evolution of classification and regression tasks and avoid one-sided dominance.

[0194] Trajectory generation and recognition module: input the fusion features into the multi-task collaborative model to generate the probability distribution of character categories and the spatiotemporal coordinate sequence of writing trajectories;

[0195] The first branch generates the probability distribution of character categories, and the specific steps are as follows:

[0196] The fusion features are nonlinearly transformed by a fully connected layer to generate a score vector :

[0197]

[0198] wherein, is the fusion feature, and are learnable parameters.

[0199] Convert to the probability distribution of character categories through a Softmax layer:

[0200]

[0201] wherein, is the probability of the m th character category, is the probability of the mThe score of each character category, is the total number of character categories.

[0202] The character with the highest probability is selected as the output result.

[0203] The second branch generates a spatiotemporal coordinate sequence, and the specific steps are as follows:

[0204] The fusion features are expanded into multi-dimensional features through a full connection layer:

[0205]

[0206] The reshaping layer reshapes the multi-dimensional features into a predicted spatiotemporal coordinate sequence :

[0207]

[0208] wherein, () is a dimension transformation operation.

[0209] In summary, the present application constructs a double-branch multi-task collaborative model, the classification branch focuses on character category prediction, the regression branch is responsible for generating a spatiotemporal coordinate sequence, and a multi-objective optimization function of fusion cross-entropy loss, trajectory distance error and shape constraint loss is designed, realizing end-to-end joint optimization of trajectory reconstruction and character recognition tasks, and effectively avoiding the error accumulation problem caused by task fragmentation.

[0210] Finally: The above only describes the preferred embodiments of the present application and is not used to limit the present application, any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A machine learning based writing trajectory optimization system, characterized in that, The method comprises the following steps: a data acquisition module: collecting first writing data of a writing process based on a preset synchronization strategy, including inertial measurement data and visual trajectory data; preprocessing the first writing data to generate second writing data, the preprocessing including dynamic length alignment, coordinate system unification, and dynamic visual compensation; a feature extraction and fusion module: generating fused features by extracting and fusing features of the second writing data through a preset feature model, the feature model being constructed based on a convolutional neural network and a long short-term memory network; a model construction module: constructing a multi-task collaborative model, the multi-task collaborative model being a double-branch architecture, different loss functions being set based on the double-branch architecture; training the multi-task collaborative model based on two training stages divided dynamically, including a first training stage for optimizing shape similarity of writing trajectories and a second training stage for strengthening distance accuracy of writing trajectory points; the double-branch architecture includes a first branch and a second branch; the first branch is a classification branch, which is composed of a fully connected layer and a Softmax layer, and is used to generate a probability distribution of character categories; the second branch is a regression branch, which is composed of a fully connected layer and a reshaping layer, and is used to generate a spatiotemporal coordinate sequence of the writing trajectory; the backbone of the multi-task collaborative model is composed of a residual connection of a convolutional neural network and a long short-term memory network; the two training stages include a first training stage and a second training stage; the dynamic division mechanism of the two training stages is as follows: the first training stage improves the overall smoothness and shape consistency of the writing trajectory by minimizing the shape constraint loss, and in the regression loss, the value of the shape loss weight is increased to amplify the influence of the shape constraint; when the first average change rate of the shape similarity loss is less than a preset first threshold, it is determined that the shape similarity has initially converged, and the second training stage is switched to; the second training stage optimizes the absolute position accuracy of the trajectory points in the spatiotemporal coordinate sequence by increasing the distance loss weight in the regression loss, and stops model training when the second average change rate is less than a preset second threshold or a preset maximum training number is reached; a trajectory generation and recognition module: inputting the fused features into the multi-task collaborative model to generate a probability distribution of character categories and a spatiotemporal coordinate sequence of the writing trajectory.

2. The system of claim 1, wherein, The feature model includes an input layer, a feature extraction layer, a time series modeling layer, a feature fusion layer, a feature enhancement layer, and an output layer; the second writing data is divided into multiple local segments through a preset sliding window; the input layer inputs the local segments; the feature extraction layer is composed of a convolutional neural network, including time series convolution and feature dimension reduction; the time series convolution uses two layers of 1D-CNN to capture the local spatiotemporal pattern of the local segment to generate abstract features; The feature dimension reduction compresses the abstract features through a pooling layer to generate compressed features ; The time sequence modeling layer is composed of a double-layer bidirectional long short-term memory network, and the second time sequence feature is generated based on the compressed feature generate a second time sequence feature .

3. The system of claim 2, wherein, The feature fusion layer fuses the compressed features and the second time sequence features to generate the fusion features, and the specific steps are as follows: 1) aligning the compression features and the second timing features on a pre-set timeline; 2) fusing the compressed features and the second temporal feature generating first fused features: in, For time steps t The first fusion feature of time, For time steps t Time-based fusion weights For element-wise multiplication, For time steps t Compression characteristics at time, For time steps t The second temporal characteristic of time; the fusion weight is obtained through an attention weight formula, as follows: wherein, with are learnable parameters, is an activation function, is a concatenation and an operation; the feature enhancement layer normalizes the first fused features; where, is the normalized time step t second fused feature at time step and are the mean and standard deviation of the fused feature at time step and are learnable parameters;​ the second fused features are compressed through a preset fully connected layer to generate third fused features; the output layer outputs all time step third fused feature sequences as the fused features.

4. The system of claim 3, wherein, The first branch will cross-entropy loss As a loss function, the difference between the probability distribution of the predicted character class and the real character is calculated, as follows: where, is the total number of character classes, is the one-hot encoding of the m th character class, is the predicted probability of the m th character class; The second branch takes a regression loss as a loss function, and the regression loss is composed of a distance-based trajectory error loss and a similarity-based shape constraint loss; Trajectory error loss adopts mean square error measures the point-to-point Euclidean distance between the predicted spatiotemporal coordinate sequence and the real spatiotemporal coordinate sequence B , which is formulated as follows: wherein, is the total number of trajectory points in the sequence of spatiotemporal coordinates, and are the true and predicted coordinates of the trajectory points in the sequence of spatiotemporal coordinates, respectively. shape constraint loss With negative Pearson correlation coefficient, the overall similarity of trajectory shape is emphasized, and the formula is: wherein, is the Pearson correlation coefficient, is the predicted spatiotemporal coordinate sequence, B is the true spatiotemporal coordinate sequence; combining the trajectory error loss and the shape constraint loss with a weighted combination to generate a regression loss : wherein, and are a distance loss weight and a shape loss weight, respectively; The classification loss and the regression loss are combined with weights to generate a multi-task total loss : wherein, with are classification and regression weights, respectively.

5. The system of claim 4, wherein, The dynamic adjustment strategy of the classification weight and the regression weight is as follows: calculating a ratio of convergence speed based on a change rate of the classification loss and the regression loss r ; When the value of the regression weight is increased, otherwise the value of the classification weight is increased; The dynamic weight average (DWA) mechanism is adopted to dynamically adjust the classification weight and the regression weight.

6. The system of claim 5, wherein, The first branch generates a probability distribution of character categories, and the specific steps are as follows: The fusion features are non-linearly transformed by the fully connected layer to generate a score vector of character classes ; transforming the score vector transforming into a probability distribution of character classes by the Softmax layer: wherein, is the probability of the m th character class, is the score of the m th character class, is the total number of character classes; The character with the highest probability is selected as the output result; The second branch generates a spatiotemporal coordinate sequence, and the specific steps are as follows: expanding the fusion feature through the fully connected layer into a multi-dimensional feature : The remodeling layer remolds the multi-dimensional features into a sequence of predicted spatio-temporal coordinates : wherein, () is a dimension transformation operation.

7. The system of claim 6, wherein, The inertial measurement data is collected by a sensor-enhanced pen based on a multi-axis sensor, including an accelerometer, a gyroscope, a magnetometer, and a force sensor; The collection of the visual trajectory data is as follows: 1) Video stream data of a camera writing process from outside to inside; 2) Compensate the image blur defects in the video stream data due to fast writing based on a preset event camera, and the specific operation is as follows: Capture the pixel-level brightness changes of the image in the video stream data asynchronously through the event camera to generate event stream data, and fuse the event stream data and the video stream data through a spatiotemporal registration formula to generate a fused image; 3) Process the fused image in the compensated video stream data based on a preset U-Net segmentation network to output a probability distribution of pen tip coordinates; 4) Take the top 20 pen tip coordinates, calculate the median as the pen tip coordinates, and generate visual trajectory data.

Images