A computer vision-based writing process analysis method

Abstract

The application discloses a writing process analysis method based on computer vision and relates to the technical field of handwriting character recognition, which comprises the following steps: performing perspective transformation on real-time writing data, obtaining a writing image frame sequence, correcting motion blur of the writing image frame sequence and positioning a pen tip coordinate by using a RAFT optical flow algorithm, forming a writing trajectory data stream, performing curvature segmentation on the writing trajectory data stream, obtaining discrete stroke segments, performing spatial topological association and geometric structure mapping on the discrete stroke segments, and generating a writing stroke topological graph. By means of the RAFT optical flow algorithm and a stroke semantic analysis model, the application improves the precision of stroke recognition and realizes efficient and accurate analysis from dynamic visual input to structured character output.

Description

Technical Field

[0001] This invention relates to the field of handwritten character recognition technology, and in particular to a method for analyzing the writing process based on computer vision. Background Technology

[0002] With the rapid development of computer vision technology, image processing-based handwritten character recognition technology has been widely applied. Traditional handwritten character recognition methods mainly rely on static image analysis, extracting features from character images and then using machine learning models for classification and recognition. However, with technological advancements, the need for understanding and analyzing the writing process is increasing, especially in areas such as educational assessment and intelligent input method optimization. In recent years, some studies have begun to incorporate the dynamic writing process into their considerations, improving recognition accuracy by tracking the writing trajectory.

[0003] Despite the progress made in existing technologies, some shortcomings remain. First, when processing writing videos against complex backgrounds, factors such as changes in lighting and shadow interference lead to poor motion blur correction, which in turn affects the accuracy of subsequent stroke recognition. Second, current technologies focus on recognizing individual characters and lack effective modeling of the topological relationships between characters during the writing process, limiting the ability to accurately recognize connected characters and characters with complex radical structures. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a computer vision-based method for analyzing the writing process to solve the problems of low stroke recognition accuracy and weak recognition ability of complex radical structures.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] This invention provides a computer vision-based method for analyzing the writing process, which includes performing perspective transformation on real-time writing data to obtain a sequence of writing image frames, and using the RAFT optical flow algorithm to perform motion blur correction and pen tip coordinate positioning on the writing image frame sequence to form a writing trajectory data stream.

[0008] Curvature segmentation is performed on the writing trajectory data stream to obtain discrete stroke fragments. Spatial topological association and geometric structure mapping are performed on the discrete stroke fragments to generate a writing stroke topology map.

[0009] The topological graph of the written strokes is input into the stroke semantic parsing model. The stroke encoding layer performs node aggregation and spatial relationship encoding, and the radical generation layer performs radical semantic modeling, outputting a written radical recognition feature map.

[0010] The structural rule matching algorithm is applied to perform relative position analysis on the feature map of written radical recognition, forming a character structure descriptor. Then, Unicode encoding and confidence evaluation are performed, and the written character parsing results are output.

[0011] As a preferred embodiment of the computer vision-based writing process analysis method of the present invention, the real-time writing data includes writing process video and writing timestamp data.

[0012] As a preferred embodiment of the computer vision-based writing process parsing method of the present invention, the step of obtaining the writing image frame sequence specifically includes the following steps:

[0013] Based on the writing timestamp data, the writing process video is time-aligned, and homography matrix transformation is performed synchronously to generate a perspective transformation frame sequence.

[0014] Histogram equalization is performed on the perspective transformation frame sequence to generate a written image frame sequence.

[0015] As a preferred embodiment of the computer vision-based writing process parsing method of the present invention, the step of forming a writing trajectory data stream specifically includes the following steps.

[0016] The RAFT optical flow algorithm is used to reconstruct the optical flow field and correct motion blur in the sequence of written image frames, forming a dense optical flow field.

[0017] Motion vector calculation is performed on the dense optical flow field to obtain pen tip coordinate positioning data, and bicubic spline interpolation is performed on the pen tip coordinate positioning data to output the writing trajectory data stream.

[0018] As a preferred embodiment of the computer vision-based writing process analysis method of the present invention, the step of obtaining discrete stroke fragments specifically includes the following steps.

[0019] The central difference derivative and Gaussian convolution are performed on the writing trajectory data stream to form a curvature parameter sequence;

[0020] Based on the curvature parameter sequence, the writing trajectory data stream is dynamically segmented to obtain discrete stroke fragments.

[0021] As a preferred embodiment of the computer vision-based writing process analysis method of the present invention, the generation of the writing stroke topology map specifically includes the following steps.

[0022] Calculate the Euclidean distance between discrete stroke segments to obtain the stroke spacing matrix; perform spatial topological association based on the stroke spacing matrix to output the stroke connection graph.

[0023] By fitting Bézier curves to the stroke connection diagram, a set of stroke geometric control points is formed. Then, the geometric structure of the stroke geometric control point set is mapped to generate a topological graph of the writing strokes.

[0024] As a preferred embodiment of the computer vision-based writing process parsing method of the present invention, the stroke semantic parsing model is specifically constructed by building a stroke encoding layer and a radical generation layer, and applying a graph attention mechanism to stack the stroke encoding layer and the radical generation layer across layers to construct the stroke semantic parsing model.

[0025] As a preferred embodiment of the computer vision-based writing process analysis method of the present invention, the step of outputting the writing radical recognition feature map specifically includes the following steps.

[0026] The topological graph of the written strokes is input into the stroke semantic parsing model. The stroke encoding layer applies graph convolution to perform node aggregation and spatial relationship encoding to generate stroke semantic vectors.

[0027] The radical generation layer uses an attention mechanism to perform semantic modeling of radicals and obtain structural representations of radicals.

[0028] Cross-modal aggregation of stroke semantic vectors and radical structure representations is performed to output a feature map for radical recognition.

[0029] As a preferred embodiment of the computer vision-based writing process parsing method of the present invention, the step of forming a character structure descriptor specifically includes the following steps:

[0030] A structural rule matching algorithm is applied to quantify the spatial relationships of the written radical recognition feature map, forming a relative position vector;

[0031] Based on the relative position vector, a geometric semantic mapping is performed on the feature map of the written radical recognition to form a character structure descriptor.

[0032] As a preferred embodiment of the computer vision-based writing process parsing method of the present invention, the output of the written character parsing result specifically includes the following steps.

[0033] Perform Unicode encoding on the character structure descriptor to obtain a set of candidate character encodings;

[0034] The TOPSIS multi-criteria algorithm is applied to evaluate the confidence of the candidate character encoding set and output the parsing results of the written characters.

[0035] The beneficial effects of this invention are as follows: By employing the motion blur correction mechanism of the RAFT optical flow algorithm and precise pen tip coordinate positioning, the problem of stroke breakage or missegmentation caused by image blurring is significantly improved, thereby enhancing the accuracy of stroke recognition. Simultaneously, by constructing a stroke semantic parsing model, joint modeling of the spatial structure and temporal relationship between strokes during writing is achieved, enhancing the semantic understanding of stroke connection relationships and complex radical combinations in connected characters, and realizing efficient and accurate parsing from dynamic visual input to structured character output. Attached Figure Description

[0036] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart of a computer vision-based method for parsing the writing process.

[0038] Figure 2 A flowchart for generating the trajectory data stream.

[0039] Figure 3 A flowchart for generating character structure descriptors.

[0040] Figure 4 A flowchart for generating discrete stroke fragments. Detailed Implementation

[0041] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0042] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0043] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0044] Reference Figures 1-4This is one embodiment of the present invention, which provides a computer vision-based method for parsing the writing process, including the following steps:

[0045] S1. Perform perspective transformation on the real-time writing data to obtain a sequence of writing image frames. Use the RAFT optical flow algorithm to perform motion blur correction and pen tip coordinate positioning on the writing image frame sequence to form a writing trajectory data stream.

[0046] The specific steps are as follows.

[0047] S1.1 Collect and preprocess real-time writing data. In specific operations, real-time writing data includes writing process video and writing timestamp data. The writing process video is collected by mobile phone camera, and the writing timestamp data is collected by inertial measurement (IMU) device.

[0048] Next, the real-time writing data is preprocessed. For the writing process video, non-writing areas are removed to eliminate background interference caused by the shooting angle. At the same time, an edge detection operator is used to enhance the contour and improve the clarity of the stroke boundaries in the writing process video. For the writing timestamp data, linear interpolation is used to unify the sampling frequency to repair sampling loss or delay. A low-pass filter is used to smooth jitter and reduce the impact of noise. Simultaneously, the RTSP timestamp synchronization protocol is used to align the data frames and output the preprocessed real-time writing data.

[0049] It should be noted that the edge detection operator is called directly through the Canny function in the OpenCV library.

[0050] S1.2. Based on the writing timestamp data, perform temporal alignment on the writing process video, synchronously execute homography matrix transformation, and generate a perspective transformation frame sequence. Specifically, based on the writing timestamp data, apply dynamic time warping for temporal alignment. Further, perform frame rate normalization on the writing process video to obtain the aligned original frame sequence; perform dual time axis interpolation compensation on the aligned original frame sequence to generate a set of synchronized video frames; perform Gaussian smoothing filtering on the set of synchronized video frames to output stable video frame data.

[0051] It should be noted that dual time-axis interpolation compensation refers to the process of resampling the original frame sequence in time and interpolating it in the spatial domain to eliminate the effects of temporal jitter and spatial misalignment.

[0052] Next, a homography matrix transformation is performed on the stable video frame data. Furthermore, eigenvalue decomposition is performed on the stable video frame data, and the top u eigenvalues ​​are extracted as key feature points. The key feature points are then expanded in their neighborhood and spatially aggregated to obtain feature point matching pairs. Geometric transformation derivation is performed on the feature point matching pairs to construct the homography matrix between adjacent frames. The homography matrix between adjacent frames describes the perspective transformation relationship between video frames and can be used to correct perspective distortion caused by the tilt of the shooting angle.

[0053] It should be noted that the geometric transformation derivation refers to the process of applying error constraints and parameter adjustments to the feature point matching pairs using the least squares method.

[0054] Based on the homography matrix between adjacent frames, bilinear interpolation is used to perform perspective transformation on the stable video frame data. Furthermore, the geometric transformation parameters of the homography matrix between adjacent frames are extracted, and the stable video frame data is resampled according to the geometric transformation parameters to generate an intermediate transformed frame sequence. The intermediate transformed frame sequence is then sharpened at the edges and enhanced in contrast to output a perspective transformed frame sequence.

[0055] S1.3. Histogram equalization is performed on the perspective transformation frame sequence to generate a writing image frame sequence. Specifically, the cvtColor function of the OpenCV library is used to convert the perspective transformation frame sequence to grayscale. Furthermore, channel separation is performed on the perspective transformation frame sequence to obtain the single-channel luminance component; normalization is performed on the single-channel luminance component to generate a preprocessed grayscale image; edge-preserving filtering is performed on the preprocessed grayscale image to output a single-channel grayscale image.

[0056] Next, the `equalizeHist` function (directly called from the OpenCV library) is applied to perform histogram equalization on the single-channel grayscale image. Further, grayscale frequency statistics are performed on the single-channel grayscale image to obtain a pixel distribution histogram. Based on the pixel distribution histogram, cumulative distribution mapping is performed on the single-channel grayscale image to achieve global contrast equalization, resulting in a preliminary equalized image. The preliminary equalized image is then divided into blocks to limit the magnitude of local contrast enhancement, obtaining locally equalized image blocks. Gamma correction is performed on the locally equalized image blocks, and the resulting image frame sequence is output.

[0057] It should be noted that cumulative distribution mapping refers to the process of probability density integration and gray value redistribution in a single-channel grayscale image; gamma correction refers to the process of nonlinear brightness adjustment in a locally equalized image.

[0058] S1.4. The RAFT optical flow algorithm is used to reconstruct the optical flow field and correct motion blur in the writing image frame sequence to form a dense optical flow field. Specifically, the motion displacement sequence in the writing image frame sequence is extracted, and the least squares method is applied to fit the motion displacement sequence to a polynomial curve. Furthermore, the motion displacement sequence is smoothed to obtain the motion trajectory parameters. Bilinear interpolation is performed on the motion trajectory parameters to generate an optimized motion curve. Nonlinear fitting is performed on the optimized motion curve to obtain the local optical flow field. The local optical flow field is then fused at multiple scales. Furthermore, the local optical flow field is decomposed hierarchically to obtain a multi-resolution optical flow field. Weighted fusion is performed on the multi-resolution optical flow field to obtain the complete optical flow field. The complete optical flow field is then vector smoothed and outlier removed to generate an optimized optical flow field.

[0059] Next, motion blur correction is performed on the optimized optical flow field. Furthermore, inter-frame motion compensation is performed on the optimized optical flow field to obtain the motion compensation field. Blur correction is performed on the motion compensation field to generate the corrected optical flow field. The corrected optical flow field is then densely encoded in two channels to output the dense optical flow field. The dense optical flow field contains the two-dimensional motion vector of each pixel, which can accurately characterize the inter-frame pixel displacement relationship and can be used for subsequent pen tip trajectory tracking.

[0060] It should be noted that inter-frame motion compensation refers to the transfer of motion vectors through the optimized optical flow field of adjacent frames to achieve inter-frame displacement compensation; fuzz correction refers to the deconvolution processing of the motion compensation field to eliminate motion blur; and dual-channel dense coding refers to the process of encoding and storing the horizontal / vertical displacement components in the corrected optical flow field separately.

[0061] S1.5. Perform motion vector calculation on the dense optical flow field to obtain pen tip coordinate positioning data, and perform bicubic spline interpolation on the pen tip coordinate positioning data to output the writing trajectory data stream. Specifically, motion vector calculation is performed on the dense optical flow field. Further, random sampling is performed on the dense optical flow field to obtain candidate motion parameters; spatiotemporal aggregation is performed on the candidate motion parameters to obtain the pen tip motion vector; the Lucas-Kanade optical flow equation is applied to calculate the displacement vector of the pen tip motion vector to generate the positioning motion vector. The specific mathematical formula is as follows.

[0062] ;

[0063] in, Indicates the positioning motion vector. Indicates the index of the random sampling point. This represents the total number of random sampling points. Indicates the first The weighting coefficients for each random sampling point Indicates the first The pen tip motion vector at each random sampling point;

[0064] It should be noted that spatiotemporal aggregation refers to the process of temporal smoothing and spatial clustering of candidate motion parameters using the sliding window averaging method; the weighting coefficient is defined based on the motion deviation rate of random sampling points, and the exemplary value range is 0.15~0.25.

[0065] The initial pen tip coordinates of the current frame are obtained by weighted averaging of the positioning motion vectors; the initial pen tip coordinates of the current frame are then temporally smoothed to obtain pen tip coordinate positioning data.

[0066] Bicubic spline interpolation is performed on the pen tip coordinate positioning data. Furthermore, control points are inserted between adjacent pen tip coordinate positioning data to obtain interpolated trajectory segments. Curve smoothing is performed on the interpolated trajectory segments to generate continuous trajectory curves. The continuous trajectory curves are uniformly resampled to maintain consistent trajectory point density, resulting in a writing trajectory data stream.

[0067] S2. Perform curvature segmentation on the writing trajectory data stream to obtain discrete stroke fragments, perform spatial topological association and geometric structure mapping on the discrete stroke fragments, and generate a writing stroke topology map.

[0068] The specific steps are as follows.

[0069] S2.1. Perform central difference differentiation and Gaussian convolution on the writing trajectory data stream to form a curvature parameter sequence. In specific operations, a sliding window is used to smooth the writing trajectory data stream to eliminate high-frequency noise and obtain smooth trajectory data; coordinate transformation is performed on the smooth trajectory data to generate smooth trajectory parameters; temporal labeling is applied to the smooth trajectory parameters to establish time correlation and obtain time-domain trajectory parameters.

[0070] The time-domain trajectory parameters are then differentiated using the central difference method. Further, the time-domain trajectory parameters are discretized to obtain discrete trajectory points. The first and second derivatives of these discrete trajectory points are then calculated to obtain the trajectory rate of change. Finally, the original curvature value of the trajectory rate of change is solved point-by-point using the curvature formula. The specific mathematical formula is as follows.

[0071] ;

[0072] in, Represents the original curvature value. Represents the rate of change of the first-order trajectory along the x-axis. express Rate of change of the second-order trajectory along the axis express Rate of change of the first-order trajectory of the axis This represents the rate of change of the second-order trajectory along the x-axis;

[0073] The original curvature values ​​are normalized to obtain standardized curvature data. The standardized curvature data is then arranged and organized in chronological order to form the original curvature sequence.

[0074] Next, Gaussian convolution is performed on the original curvature sequence. Then, a weighted average is applied to the original curvature sequence to smooth the curvature fluctuations and obtain an intermediate curvature sequence. Bilateral filtering is then performed on the intermediate curvature sequence to eliminate local noise interference and generate an optimized curvature sequence. Finally, a linear transformation is applied to normalize the optimized curvature sequence and output the curvature parameter sequence.

[0075] By employing four steps—sliding window, central difference differentiation, curvature formula calculation, and Gaussian convolution—the obtained curvature parameter sequence is ensured to accurately reflect the geometric characteristics of the writing trajectory while also exhibiting robustness against noise interference.

[0076] S2.2. Based on the curvature parameter sequence, the writing trajectory data stream is dynamically segmented to obtain discrete stroke fragments. Specifically, an extremum detection algorithm is used to extract effective local extremum points from the curvature parameter sequence. Furthermore, curvature change detection is performed on the curvature parameter sequence to obtain change feature points. Differential processing is performed on the change feature points to obtain candidate extremum points. The candidate extremum points are then subject to amplitude range limitation and time interval constraint to obtain effective local extremum points.

[0077] It should be noted that the amplitude range limitation refers to constraining the candidate extreme points within a fixed range (e.g., 0~1 mm) to avoid interference from outliers; the time interval constraint refers to the process of time-series alignment and minimum interval (e.g., 50 milliseconds) control of the candidate extreme points to ensure the uniformity of the distribution of effective local extreme points.

[0078] Next, the stroke turning points in the valid local extreme points are identified by the turning point threshold: when the valid local extreme point exceeds the turning point threshold, it is determined to be a valid turning point and is retained; when the amplitude of the extreme point is lower than the turning point threshold, it is determined to be an invalid fluctuation and is removed.

[0079] It should be noted that the turning point threshold is defined based on the curvature statistical characteristics of historical stroke turning points, with an exemplary value range of 0.15 to 0.35.

[0080] The effective turning points are combined with the trajectory change rate for secondary verification. For example, the curvature change gradient of the effective turning points is extracted. When the curvature change gradient is greater than the trajectory change rate, it is defined as a true turning point; when the curvature change gradient is less than the trajectory change rate, it is defined as a false turning point (such as temporary fluctuations caused by writing jitter).

[0081] Based on the actual turning points, the writing trajectory data stream is dynamically segmented. Furthermore, the writing trajectory data streams of adjacent actual turning points are divided into independent trajectory segments. The segmented independent trajectory segments are smoothly connected to generate continuous stroke fragments, and the continuous stroke fragments are discretized to form discrete stroke fragments.

[0082] Through three steps—mean filtering, local extreme point location, and trajectory change rate verification—it is ensured that the resulting discrete stroke fragments can accurately reflect the natural stroke structure of writing and adapt to individual differences in writing speed and style.

[0083] S2.3. Calculate the Euclidean distance for discrete stroke segments to obtain the stroke spacing matrix; perform spatial topological association based on the stroke spacing matrix to output a stroke connection graph. Specifically, for each discrete stroke segment, extract coordinate values ​​and calculate the Euclidean distance, outputting the stroke spacing value. The specific mathematical format is as follows.

[0084]

[0085] in, Indicates the stroke spacing value. Indicates the second discrete stroke segment axis coordinate values, Represents the first discrete stroke segment axis coordinate values, Indicates the second discrete stroke segment axis coordinate values, Represents the first discrete stroke segment Axis coordinate values;

[0086] Next, matrix filling is performed on the stroke spacing values ​​to obtain the stroke spacing matrix, and the stroke spacing matrix is ​​binarized: the distance elements of the stroke spacing matrix are extracted, the distance elements less than the connection threshold are set to 1 (indicating that there is a connection between the strokes), and the distance elements greater than the connection threshold are set to 0 (indicating that there is no connection between the strokes), and the binary connection matrix is ​​output.

[0087] It should be noted that the connection threshold is defined based on the spatial distribution characteristics of the historical stroke spacing matrix, with an exemplary value range of 3.5~5.2 mm.

[0088] A depth-first search algorithm is used to perform spatial topological association on the binary connectivity matrix. Further, a neighborhood traversal is performed on the binary connectivity matrix to obtain a neighborhood connectivity matrix, and topological expansion is performed on the neighborhood connectivity matrix to generate connected components. Spatial clustering is then performed on the connected components to obtain their spatial location distribution. Based on the spatial location distribution, the connected components are grouped, with connected components of the same spatial location distribution grouped into the same connectivity group. Strokes within each connectivity group are then topologically connected according to the principle of consistent writing direction, and a stroke connectivity graph is output.

[0089] It should be noted that spatial clustering refers to the process of measuring distance and dividing regions of connected components; the principle of consistency in writing direction is defined based on the angle between the tangent directions of the stroke endpoints.

[0090] S2.4. Perform Bézier curve fitting on the stroke connection diagram to form a set of stroke geometric control points. Perform geometric structure mapping on the set of stroke geometric control points to generate a writing stroke topology diagram. In specific operations, extract the curvature change feature points in the stroke connection diagram and perform density clustering to obtain the fitting control points; perform curve fitting on the fitting control points to generate a G1 continuous fitting curve; perform uniform resampling on the G1 continuous fitting curve to obtain an optimized control point set, and perform smoothness optimization on the optimized control point set to output the stroke geometric control point set.

[0091] It should be noted that density clustering refers to the process of neighborhood search and weighted aggregation of feature points with curvature changes to ensure the rationality of the spatial distribution of fitting control points.

[0092] Next, the geometric structure of the stroke geometric control point set is mapped. Further, according to the topological connection rules, the stroke geometric control point set is projected by principal components to obtain geometric feature vectors. Eigenvalue decomposition is performed on the geometric feature vectors, and the first h feature values ​​are extracted, sorted by dimension reduction, and a spatial hierarchical index is constructed. According to the spatial hierarchical index, the hierarchical relationship is mapped on the geometric feature vectors to obtain a preliminary topological map. The preliminary topological map is then rendered and refined to output the writing stroke topological map.

[0093] It should be noted that the topological connectivity rule is based on the Euclidean distance and tangent direction definition of the stroke geometric control point set; principal component projection refers to the process of performing feature basis transformation on the stroke geometric control point set.

[0094] S3. Input the writing stroke topology map into the stroke semantic parsing model. The stroke encoding layer performs node aggregation and spatial relationship encoding, the radical generation layer performs radical semantic modeling, and outputs the writing radical recognition feature map.

[0095] The specific steps are as follows.

[0096] S3.1 Construct and train a stroke semantic parsing model. Specifically, in the PyTorch framework, a graph convolutional network is called using the `nn.Module` parameter, with the input dimension set to 64, the output dimension to 128, and the activation function set to ReLU. A Dropout layer is then added after the graph convolutional network for regularization to prevent overfitting, completing the stroke encoding layer. The `get_encoder` function is used to call the Transformer encoder architecture, embedding an attention mechanism to model radical semantics and capture long-range dependencies between strokes. The embedding dimension of the Transformer encoder architecture is set to 128, the number of attention heads to 8, and the feedforward network dimension to 512. BatchNorm1d is then added after the Transformer encoder architecture for feature normalization, completing the radical generation layer.

[0097] A graph attention mechanism is used to perform feature interaction between the stroke encoding layer and the radical generation layer to obtain cross-level fusion features. Residual connections are then made on the cross-level fusion features to generate a hybrid semantic representation. A fully connected layer is used to linearly project the hybrid semantic representation to obtain radical recognition features. The radical recognition features are then probability-normalized using a softmax function to obtain radical category weights. Based on the radical category weights, the stroke encoding layer and the radical generation layer are stacked across layers through skip connections to complete the construction of the stroke semantic parsing model.

[0098] Next, the stroke semantic parsing model is trained. Further, the historical stroke topology map is divided into a sample set, a training set, and a validation set. On the sample set, random affine transformations are used for data augmentation, and a data loader is used to batch read data to form augmented training samples. On the training set, the Adam optimizer is used to backpropagate the augmented training samples, and a cosine annealing device is applied simultaneously for dynamic learning rate adjustment to obtain validation metrics. On the validation set, the cross-entropy loss function is applied to quantify the validation metrics, obtaining the validation loss value. When the validation loss value exceeds the convergence threshold for several consecutive rounds (e.g., 5 times), training terminates, and the trained stroke semantic parsing model is output simultaneously.

[0099] It should be noted that the convergence threshold is defined based on the moving average rate of change of historical validation loss values, with an exemplary range of 0.02 to 0.06.

[0100] S3.2 The stroke encoding layer uses graph convolution to perform node aggregation and spatial relationship encoding to generate stroke semantic vectors. Specifically, the writing stroke topology graph is input into the stroke encoding layer of the stroke semantic parsing model through the graphical interface of the OpenCV library, and node aggregation is performed through two layers of graph convolutional networks. Further, the first layer uses a 64×128 graph convolution to perform neighborhood feature aggregation on the writing stroke topology graph to obtain the intermediate representation of the nodes. The ReLU activation function is applied to the intermediate representation of the nodes to perform nonlinear mapping and output the activated trajectory node features. The second layer uses a 128×128 graph convolution to perform weighted aggregation on the activated trajectory node features to generate higher-order feature representations. The Dropout layer is then applied to randomly discard the higher-order feature representations to generate regularized features.

[0101] Next, spatial relationship encoding is performed on the regularized features. Further, sinusoidal position embedding is applied to the regularized features to obtain spatial relationship features. Local receptive field expansion is performed on the spatial relationship features to obtain multi-scale encoded features, which are then integrated. Further, the multi-scale encoded features are dynamically weighted to obtain weighted encoded features. Nonlinear transformation is applied to the weighted encoded features to generate higher-order encoded feature representations. These higher-order encoded feature representations are then fused to obtain composite feature representations. Global average pooling is used to compress the composite feature representations to generate stroke semantic vectors. These stroke semantic vectors not only capture local stroke features but also preserve global topological relationships, improving the completeness and discriminative power of stroke representations.

[0102] It should be noted that sinusoidal position embedding refers to the process of periodically encoding the positional information of regularized features using sine / cosine functions.

[0103] S3.3 The radical generation layer performs semantic modeling of radicals through an attention mechanism to obtain the structural representation of radicals. Specifically, an affine transformation function is applied to linearly project the stroke semantic vector, decomposing the stroke semantic vector into a query vector, a key vector, and a value vector. Dot product scaling is then performed on the query vector, key vector, and value vector to obtain attention scores. These attention scores are then normalized to obtain the association weights between strokes. The specific mathematical formula is as follows.

[0104] ;

[0105] in, Indicates the weight of the relationship between strokes. Represents the query vector. Represents the key vector. This indicates the transpose operation. Represents a value vector. Indicates the scaling factor;

[0106] It should be noted that the scaling factor is defined based on the square root of the key vector dimension, with an exemplary range of 8 to 64.

[0107] Next, based on the correlation weights between strokes, weighted aggregation is performed on the stroke semantic vectors to obtain attention-weighted stroke features. These attention-weighted stroke features are then concatenated across stroke heads to obtain high-dimensional radical fusion features. Linear transformation and feature dimensionality reduction are then applied to these high-dimensional radical fusion features to generate a global and local radical semantic representation. Finally, residual connections and layer normalization are applied to the global and local radical semantic representation to ensure feature stability and output the radical structure representation.

[0108] The radical structure representation can not only characterize the spatial topological relationship between strokes, but also model the semantic combination rules of radicals, realizing a hierarchical mapping from geometric features to semantic structure.

[0109] S3.4. Cross-modal aggregation of stroke semantic vectors and radical structure representations is performed to obtain a writing radical recognition feature map. Specifically, a fully connected layer is used to uniformly map the stroke semantic vectors and radical structure representations to the same feature space to achieve dimension alignment. A cross-attention mechanism is used to perform cross-modal aggregation of the aligned stroke semantic vectors and radical structure representations. Furthermore, the aligned stroke semantic vectors and radical structure representations are split into multiple heads to obtain query features, key-direction features, and value features. Attention weighting is applied to the query features, key-direction features, and value features to obtain cross-modal interaction features. Channel concatenation of the cross-modal interaction features is then performed to generate a three-channel hybrid vector.

[0110] A gating mechanism is used to regulate the information of the three-channel hybrid vector. Further, the reset gate performs feature masking on the three-channel hybrid vector to obtain the reset features; the update gate performs selective fusion and nonlinear activation on the reset features to form the gated feature representation; the output gate performs linear transformation and dimensionality compression on the gated feature representation to output the radical recognition feature map.

[0111] It should be noted that feature masking refers to the process of weighting and masking features in a channel blending vector.

[0112] S4. Apply the structural rule matching algorithm to perform relative position analysis on the written radical recognition feature map, form a character structure descriptor, perform Unicode encoding and confidence evaluation, and output the written character parsing result.

[0113] The specific steps are as follows.

[0114] S4.1. Quantify the spatial relationship of the handwritten radical recognition feature map using the structural rule matching algorithm to form a relative position vector. Specifically, in the specific operation, define a radical spatial relationship template based on the topological features of Chinese character glyphs, including typical layout patterns of left-right structure (such as "好"), up-down structure (such as "字"), and enclosed structure (such as "国");

[0115] According to the radical spatial relationship template, quantify the spatial relationship of the handwritten radical recognition feature map. Further, extract the geometric parameters of the radicals in the handwritten radical recognition feature map, perform coordinate transformation on the radical geometric parameters to obtain the standardized radical position parameters; decompose the standardized radical position parameters in the horizontal and vertical directions to obtain the radical horizontal component and the radical vertical component; perform area ratio conversion on the radical horizontal component and the radical vertical component to generate the overlapping area ratio; perform weighted fusion on the radical horizontal component, the radical vertical component, and the overlapping area ratio to obtain the spatial relationship feature;

[0116] Finally, perform temporal alignment and similarity comparison between the spatial relationship feature and the radical spatial relationship template to obtain a matching score. When the matching score > matching threshold (such as 0.7), it is determined as a successful match, and the successfully matched spatial relationship feature is structurally encoded to output the relative position vector;

[0117] It should be noted that the matching threshold is defined based on the distribution characteristics of historical matching scores. Structural encoding refers to the process of dimensional reduction and feature sorting for the spatial relationship feature.

[0118] S4.2. Perform geometric semantic mapping on the handwritten radical recognition feature map according to the relative position vector to form a character structure descriptor. Specifically, perform residual convolution on the handwritten radical recognition feature map to obtain the deep visual feature; perform multi-layer perceptron transformation on the relative position vector to obtain a high-dimensional position encoding; integrate the deep visual feature and the high-dimensional position encoding to generate a visual-position joint representation;

[0119] It should be noted that the multi-layer perceptron transformation refers to the process of non-linearly mapping the relative position vector using a fully connected layer and an activation function.

[0120] Use spatial pyramid pooling (SPP) to perform geometric semantic mapping on the visual-position joint representation. Further, perform multi-scale pooling on the visual-position joint representation to obtain local spatial features; perform linear projection on the local spatial features to map the local spatial features to the global semantic space to generate geometric semantic features; perform character structure conversion on the geometric semantic features to obtain a character-level semantic vector, and perform standardization processing and structural encapsulation on the character-level semantic vector to output the character structure descriptor.

[0121] S4.3. Encode the character structure descriptor using Unicode to obtain a set of candidate character codes. In practice, a Chinese character encoding mapping table is defined based on the statistical feature distribution of historical character structure descriptors, which includes Unicode code points and corresponding Chinese character structure feature templates.

[0122] Next, the character structure descriptor is Unicode encoded according to the Chinese character encoding mapping table. Furthermore, the character structure descriptor is compared with the Chinese character structure feature template in the Chinese character encoding mapping table. If the similarity reaches a reasonable range (e.g., 0.7~0.9), the corresponding Unicode code point is taken as a candidate code point. The candidate code points are sorted in descending order, and the top N (e.g., N=5) candidate code points are integrated to form a candidate character encoding set.

[0123] It should be noted that the reasonable range is defined based on the typical fluctuation range of historical comparison similarity.

[0124] S4.4. The TOPSIS multi-criteria algorithm is applied to evaluate the confidence of the candidate character encoding set and output the written character parsing results. Specifically, the candidate character encoding set is boundary-localized. Further, a feature space traversal is performed on the candidate character encoding set to obtain the character feature distribution. Neighborhood aggregation is performed on the character feature distribution to generate boundary feature clusters. Center point sampling is performed on the boundary feature clusters to obtain boundary reference values. The maximum value of the boundary reference value is taken as the ideal solution, and the minimum value is taken as the negative ideal solution. The entropy weight method is applied to calculate the confidence of the ideal and negative ideal solutions, and the confidence evaluation value is output. The specific mathematical formula is as follows.

[0125] ;

[0126] in, Indicates the index of the candidate character encoding set. Indicates the first Confidence assessment values ​​for candidate character encodings; Indicates the first The distance between each candidate character encoding and the ideal solution Indicates the first The distance between the candidate character encoding and the negative ideal solution. This represents the nonlinear adjustment coefficient;

[0127] It should be noted that the nonlinear adjustment coefficient is defined based on the convergence rate of historical confidence assessment values, and the exemplary value range is 1.2 to 2.5.

[0128] Based on the confidence score, the candidate character encoding set is effectively screened. Further, the relative positions of the candidate character encoding set are compared. Then, the geometric features of the radicals in the candidate character encoding set are extracted and their spatial relationships are quantified to obtain the relative structural positions. These relative structural positions are then compared for similarity to obtain the structural closeness. When the structural closeness of the candidate character encoding set is greater than the confidence score, it is defined as a valid parsing result. The valid parsing results are then structurally reorganized to output the written character parsing result.

[0129] This embodiment also provides a computer device applicable to the computer vision-based writing process parsing method, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the computer vision-based writing process parsing method proposed in the above embodiment.

[0130] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0131] This embodiment also provides a storage medium on which a computer program is stored. When executed by a processor, the program implements the computer vision-based writing process parsing method proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0132] In summary, this invention significantly improves the accuracy of stroke recognition by combining the motion blur correction mechanism of the RAFT optical flow algorithm with precise pen tip coordinate positioning, thereby addressing the problem of stroke breakage or missegmentation caused by image blur. Simultaneously, by constructing a stroke semantic parsing model, it achieves joint modeling of the spatial structure and temporal relationships between strokes during writing, enhancing the semantic understanding of stroke connection relationships and complex radical combinations in connected characters. This enables efficient and accurate parsing from dynamic visual input to structured character output.

[0133] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for parsing the writing process based on computer vision, characterized in that: include, The real-time writing data is subjected to perspective transformation to obtain a sequence of writing image frames. The RAFT optical flow algorithm is used to correct motion blur and locate pen tip coordinates in the sequence of writing image frames to form a writing trajectory data stream. Curvature segmentation is performed on the writing trajectory data stream to obtain discrete stroke fragments. Spatial topological association and geometric structure mapping are performed on the discrete stroke fragments to generate a writing stroke topology map. The stroke topology graph is input into the stroke semantic parsing model. The specific construction process is as follows: a stroke encoding layer and a radical generation layer are built, and a graph attention mechanism is applied to stack the stroke encoding layer and the radical generation layer across layers to build the stroke semantic parsing model. The stroke encoding layer is built by calling a graph convolutional network, and the radical generation layer is built by the Transformer encoder architecture and attention mechanism. The stroke encoding layer performs node aggregation and spatial relationship encoding, while the radical generation layer performs radical semantic modeling and outputs a feature map for written radical recognition. Specifically, this includes the following steps: The topological graph of the written strokes is input into the stroke semantic parsing model. The stroke encoding layer applies graph convolution to perform node aggregation and spatial relationship encoding to generate stroke semantic vectors. The radical generation layer uses an attention mechanism to perform semantic modeling of radicals and obtain structural representations of radicals. The stroke semantic vectors and radical structure representations are aggregated across modalities to output a writing radical recognition feature map. The stroke semantic vectors and radical structure representations are uniformly mapped to the same feature space through a fully connected layer to achieve dimension alignment. A cross-attention mechanism is used to aggregate the aligned stroke semantic vectors and radical structure representations across modalities. The structural rule matching algorithm is applied to perform relative position analysis on the feature map of written radical recognition, forming a character structure descriptor. Then, Unicode encoding and confidence evaluation are performed, and the written character parsing results are output.

2. The computer vision-based writing process analysis method as described in claim 1, characterized in that: The real-time writing data includes video of the writing process and writing timestamp data.

3. The computer vision-based writing process analysis method as described in claim 2, characterized in that: The acquisition of the writing image frame sequence specifically includes the following steps. Based on the writing timestamp data, the writing process video is time-aligned, and homography matrix transformation is performed synchronously to generate a perspective transformation frame sequence. Histogram equalization is performed on the perspective transformation frame sequence to generate a written image frame sequence.

4. The computer vision-based writing process parsing method as described in claim 1, characterized in that: The formation of the writing trajectory data stream specifically includes the following steps. The RAFT optical flow algorithm is used to reconstruct the optical flow field and correct motion blur in the sequence of written image frames, forming a dense optical flow field. Motion vector calculation is performed on the dense optical flow field to obtain pen tip coordinate positioning data, and bicubic spline interpolation is performed on the pen tip coordinate positioning data to output the writing trajectory data stream.

5. The computer vision-based writing process parsing method as described in claim 1, characterized in that: The process of obtaining discrete stroke fragments specifically includes the following steps. The central difference derivative and Gaussian convolution are performed on the writing trajectory data stream to form a curvature parameter sequence; Based on the curvature parameter sequence, the writing trajectory data stream is dynamically segmented to obtain discrete stroke fragments.

6. The computer vision-based writing process parsing method as described in claim 5, characterized in that: The process of generating the writing stroke topology map specifically includes the following steps. Euclidean distance is calculated for discrete stroke segments to obtain the stroke spacing matrix; Based on the stroke spacing matrix, perform spatial topological association and output the stroke connection graph; By fitting Bézier curves to the stroke connection diagram, a set of stroke geometric control points is formed. Then, the geometric structure of the stroke geometric control point set is mapped to generate a topological graph of the writing strokes.

7. The computer vision-based writing process parsing method as described in claim 1, characterized in that: The process of forming a character structure descriptor specifically includes the following steps. A structural rule matching algorithm is applied to quantify the spatial relationships of the written radical recognition feature map, forming a relative position vector; Based on the relative position vector, a geometric semantic mapping is performed on the feature map of the written radical recognition to form a character structure descriptor.

8. The computer vision-based writing process parsing method as described in claim 1, characterized in that: The output of the written character parsing result specifically includes the following steps. Perform Unicode encoding on the character structure descriptor to obtain a set of candidate character encodings; The TOPSIS multi-criteria algorithm is applied to evaluate the confidence of the candidate character encoding set and output the parsing results of the written characters.

Images