A cloud platform-based language auxiliary teaching method and system

Abstract

The application relates to a cloud platform-based Chinese auxiliary teaching method and system. The method comprises the following steps: generating an enhanced document data structure by performing handwriting-text cross-modal space-time alignment and semantic binding on dynamic handwriting data of students in the writing process and static text data after the writing is completed; mapping the meta-cognition features of each character in the enhanced document data structure into corresponding meta-cognition state labels to obtain a labeled character sequence, combining a writing quality score, constructing a directed acyclic causal graph and performing parameter learning to generate causal effect parameters; and based on the causal effect parameters, tracing specific attribution paths through counterfactual reasoning to finally generate an individualized tutoring report. The method can cross-modally associate handwriting tracks and text semantics and can perform an interpretable modeling on the writing cognitive process of students, thereby significantly improving the diagnosticity and teaching value of the Chinese auxiliary teaching system.

Description

Technical Field

[0001] This invention belongs to the field of Chinese language teaching assistance technology, and in particular relates to a Chinese language teaching assistance method and system based on a cloud platform. Background Technology

[0002] With the rapid development of intelligent education technology, intelligent essay correction systems have been widely used in the field of Chinese language teaching assistance. These systems typically use optical character recognition technology to convert handwritten essays into electronic text, and then use natural language processing models to automatically evaluate aspects such as grammar, logic, and text structure, thereby assisting teachers in completing essay correction work.

[0003] Most existing intelligent essay grading systems only analyze the final essays submitted by students, processing static, completed texts. In traditional Chinese language teaching, teachers also grade essays based on the final draft, focusing on the completeness of the structure, the accuracy of the language, and the richness of the content.

[0004] However, existing technologies overlook the crucial characteristic of writing itself as a dynamic cognitive process. From the moment a student puts pen to paper until the essay is completed, the process involves numerous actions such as hesitation, pauses, corrections, rewriting, and adjustments to the thought process. This procedural handwriting data carries rich signals reflecting metacognitive states such as difficulties in vocabulary retrieval, syntactic organization obstacles, and writer's block. The lack of a solution to deeply integrate and causally link the dynamic handwriting data during the writing process with the final static text content prevents the system from revealing the implicit cognitive activities of students during writing and makes it difficult to provide diagnostic and interpretable personalized guidance. Summary of the Invention

[0005] Therefore, it is necessary to provide a Chinese language teaching aid that can cross-modal correlate handwriting trajectories with text semantics and interpretably model the students' writing cognitive process, in order to address the above-mentioned technical problems.

[0006] Firstly, this application provides a cloud-based method for assisting Chinese language teaching, including:

[0007] The system acquires dynamic handwriting data during the writing process and static text data after the writing is completed, and performs handwriting-text cross-modal spatiotemporal alignment and semantic binding on the static text data and dynamic handwriting data to generate an enhanced document data structure.

[0008] The metacognitive features of each character in the enhanced document data structure are mapped to the corresponding metacognitive state labels to obtain a labeled character sequence; wherein, the metacognitive features include point-level micro features and character-level meso features; the character is the smallest semantic unit in static text data;

[0009] Using characters in a labeled character sequence as nodes, a directed acyclic causal graph is constructed, and parameters of the directed acyclic causal graph are learned based on writing quality scores to generate causal effect parameters.

[0010] Based on the causal effect parameters, counterfactual reasoning is performed on each character to generate the causal effect strength. Based on the causal effect strength, reverse path tracing is performed in the directed acyclic causal graph to obtain the attribution path chain.

[0011] Based on metacognitive state labels, causal effect strength, and attribution path chain, a Chinese language tutoring report is generated.

[0012] Furthermore, dynamic handwriting data during the writing process and static text data after completion are acquired. Handwriting-text cross-modal spatiotemporal alignment and semantic binding are then performed on the static text data and dynamic handwriting data to generate an enhanced document data structure, including:

[0013] Acquire dynamic handwriting data of students during the writing process and images of the final draft pages after the writing is completed; the dynamic handwriting data includes time sequence, coordinates, pressure, and pen posture;

[0014] Derived features are extracted from dynamic handwriting data to obtain derived features, and the dynamic handwriting data and derived features are fused to obtain enhanced handwriting time series data; among them, the derived features include instantaneous velocity, instantaneous acceleration, pressure change gradient and pen pause time;

[0015] Perform text recognition on the final draft page image to generate static text data;

[0016] We perform cross-modal spatiotemporal alignment and semantic binding of handwriting-text data on static text data and enhanced handwriting time-series data to generate enhanced document data structures.

[0017] Furthermore, handwriting-text cross-modal spatiotemporal alignment and semantic binding are performed on static text data and enhanced handwriting time-series data to generate an enhanced document data structure, including:

[0018] Based on the coordinates of the center point of the minimum bounding rectangle of at least three characters in the static text data and the physical coordinates of the corresponding handwriting points in the enhanced handwriting time series data, the affine transformation matrix is ​​obtained by fitting the least squares method.

[0019] Based on the affine transformation matrix, the physical coordinates of all handwriting sampling points in the enhanced handwriting time series data are transformed to the image pixel coordinate system of the static text data to obtain the aligned handwriting sampling point sequence.

[0020] A dynamic time warping algorithm is used to coarsely align the handwriting sampling point sequence and the character sequence in the static text data to obtain the handwriting sampling point index range corresponding to each character.

[0021] The time and number of aligned handwriting sampling points within the index interval of each handwriting sampling point are normalized to obtain normalized time points.

[0022] Writing dynamics parameters are extracted at each normalized time point and summarized to obtain the writing dynamics parameter sequence for the corresponding character; among which, the writing dynamics parameters include instantaneous velocity, pressure, acceleration and pressure gradient value;

[0023] Based on the intersection-union ratio of the bounding box of each stroke in the aligned handwriting sampling point sequence and the minimum bounding rectangle of each character in the static text data, combined with the preset tampering judgment threshold, the tampering behavior is judged and the judgment result is obtained.

[0024] The sequence of writing dynamics parameters and the judgment results for each character are integrated into the character structure of the static text data to obtain an enhanced document data structure.

[0025] Furthermore, derived features are extracted from the dynamic handwriting data to obtain derived features. The dynamic handwriting data and derived features are then fused to obtain enhanced handwriting time-series data, including:

[0026] Each stroke in the dynamic handwriting data is filtered to obtain smooth handwriting sampling points. All smooth handwriting sampling points are arranged in chronological order to obtain a smooth handwriting sampling point sequence.

[0027] Based on the coordinate difference and first time difference between adjacent sampling points in the smooth handwriting sampling point sequence, the instantaneous velocity and instantaneous acceleration of each sampling point are calculated, and the instantaneous velocity and instantaneous acceleration are arranged in time order to obtain the instantaneous velocity sequence and instantaneous acceleration sequence respectively;

[0028] Based on the pressure difference and second time difference between adjacent sampling points in the smooth handwriting sampling point sequence, the pressure change gradient is calculated, and the pressure change gradient is arranged in time order to obtain the pressure change gradient sequence.

[0029] Based on the smooth handwriting sampling point sequence, the pen pause time is extracted from the third time difference between the end time of the current stroke and the start time of the next stroke, and the pen pause time is arranged in chronological order to obtain the pen pause time sequence.

[0030] The time axis is aligned and stitched together for the smooth handwriting sampling point sequence, timestamp sequence, instantaneous velocity sequence, instantaneous acceleration sequence, pressure change gradient sequence, and pen pause time sequence for each stroke to obtain the strong handwriting time series data for each sampling point.

[0031] Furthermore, the metacognitive features of each character in the enhanced document data structure are mapped to corresponding metacognitive state labels, resulting in a labeled character sequence, including:

[0032] Based on each character and the sequence of writing dynamics parameters of the character in the enhanced document data structure, point-level micro-features are calculated at each normalized time point; among which, point-level micro-features include writing fluency score, pressure stability index, pen suspension pause duration before writing and pen suspension pause duration after writing.

[0033] Statistical aggregation of point-level micro-features at all normalized time points within the same character yields character-level meso-features; among which, character-level meso-features include mean pressure, mean velocity, mean absolute value of acceleration, pressure variance, velocity variance, velocity coefficient of variation, number of erasures, interquartile range of pressure, and average erasure time.

[0034] Each character-level mesoscopic feature is standardized to obtain standardized character-level features, and the standardized character-level features are then aggregated to obtain a standardized feature vector.

[0035] The standardized feature vector is input into the machine learning classifier, which outputs metacognitive state labels and confidence scores of the metacognitive state labels.

[0036] The metacognitive state label and confidence score of each character are arranged in the order of the original text to obtain a labeled character sequence.

[0037] Secondly, this application also provides a cloud-based Chinese language teaching assistance system, including:

[0038] The data acquisition module is used to acquire dynamic handwriting data of students during the writing process and static text data after the writing is completed. It performs handwriting-text cross-modal spatiotemporal alignment and semantic binding on the static text data and dynamic handwriting data to generate an enhanced document data structure.

[0039] The tag generation module is used to map the metacognitive features of each character in the enhanced document data structure to the corresponding metacognitive state label, resulting in a labeled character sequence; wherein, the metacognitive features include point-level micro features and character-level meso features;

[0040] The parameter learning module is used to construct a directed acyclic causal graph using characters in a labeled character sequence as nodes, and to learn the parameters of the directed acyclic causal graph based on writing quality scores to generate causal effect parameters.

[0041] The parameter analysis module is used to perform counterfactual reasoning on each character based on causal effect parameters, generate causal effect strength, and perform reverse path tracing in the directed acyclic causal graph based on causal effect strength to obtain the attribution path chain.

[0042] The report generation module is used to generate Chinese language tutoring reports based on metacognitive state labels, causal effect strength, and attribution path chains.

[0043] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to implement any of the cloud platform-based Chinese language teaching methods described in the embodiments of this application.

[0044] Fourthly, this application also provides a computer-readable storage medium storing at least one piece of program code, which is loaded and executed by a processor to implement a cloud-based language teaching aid method as described in any of the embodiments of this application.

[0045] The aforementioned cloud-based Chinese language teaching assistance method and system semantically binds temporal handwriting data and the final text through cross-modal spatiotemporal alignment of handwriting and text; extracts metacognitive labels reflecting students' cognitive states such as vocabulary extraction and syntactic organization based on point-level and character-level features; quantifies the causal effect of each character's metacognitive features on writing quality using a causal graph model; traces specific attribution paths through counterfactual reasoning; and finally generates interpretable, evidence-based personalized tutoring reports. This effectively addresses the problem that existing technologies cannot reveal students' implicit cognitive activities during the writing process, enabling teachers to accurately pinpoint students' difficulties in specific cognitive stages such as vocabulary extraction, syntactic organization, or text structure, significantly enhancing the diagnostic and pedagogical value of the Chinese language teaching assistance system. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating a cloud-based language teaching aid method in one embodiment.

[0048] Figure 2 This is a flowchart illustrating the steps of obtaining dynamic handwriting data of students during the writing process and static text data after the writing is completed in one embodiment, and performing handwriting-text cross-modal spatiotemporal alignment and semantic binding on the static text data and dynamic handwriting data to generate an enhanced document data structure.

[0049] Figure 3 This is a schematic diagram of the structure of a cloud-based Chinese language teaching assistance system in one embodiment. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] In one embodiment, a cloud-based method for assisting Chinese language teaching is provided. This embodiment illustrates the application of this method to a terminal, but it is understood that the method can also be applied to a server, or to a system including both a terminal and a server, and is implemented through the interaction between the terminal and the server. Figure 1 As shown, in this embodiment, the method includes the following steps:

[0052] Step S101: Obtain dynamic handwriting data of students during the writing process and static text data after the writing is completed, and perform handwriting-text cross-modal spatiotemporal alignment and semantic binding on the static text data and dynamic handwriting data to generate an enhanced document data structure.

[0053] For example, dynamic handwriting data generated by students during writing can be collected at a fixed sampling rate using an electromagnetic induction digital writing tablet. This data includes the coordinates of each sampling point, the pen tip pressure value, the pen tilt angle, and a millisecond-level timestamp. All sampling points form the original handwriting sequence in chronological order. The ink status on the writing tablet is monitored. When the student finishes writing and triggers a submission operation, or when no new strokes are input within a preset time, the rendered image of all ink on the current tablet is captured to obtain the final draft page image. An OCR engine deployed on a cloud platform is invoked to perform character detection and character recognition on the final draft page image, obtaining the text content of each character and the coordinates of the four vertices of its minimum bounding rectangle in the image coordinate system, thereby obtaining static text data including character sequence and spatial location information. Handwriting-text cross-modal spatiotemporal alignment and semantic binding are performed on the static text data and dynamic handwriting data to generate an enhanced document data structure.

[0054] Among them, OCR (Optical Character Recognition) refers to the technology of automatically recognizing and converting text content in an image into editable and searchable text data. The OCR engine is the core software component for realizing this technology. Electromagnetic induction digital writing tablet is a digital writing device that uses the principle of electromagnetic induction to realize handwriting input. Its working method is to generate an alternating electromagnetic field through an array of electromagnetic coils inside the tablet. When a specially designed electromagnetic pen is close to the tablet surface, the coil inside the pen induces a current and feeds back a signal, thereby accurately locating information such as the position, pressure, and tilt of the pen tip, realizing handwriting input and drawing functions without batteries, with high precision and high response speed. Preset duration refers to a period of time set in advance. Character detection refers to locating all text areas in an image, distinguishing between text and non-text, and selecting the location and range of the character. Character recognition refers to decoding the detected text areas character by character / line by line, converting the pixel text in the image into editable text characters.

[0055] Step S102: Map the metacognitive features of each character in the enhanced document data structure to the corresponding metacognitive state label to obtain a labeled character sequence; wherein, the metacognitive features include point-level micro features and character-level meso features; the character is the smallest semantic unit in static text data.

[0056] In this context, "mapping" refers to the process by which an element in one set is transformed into / points to a unique element in another set according to fixed rules, i.e., a one-to-one or many-to-one correspondence.

[0057] For example, each character in the enhanced document data structure is used as a processing unit. Micro-features at each normalized time point are extracted from the sequence of writing dynamics parameters bound to that character. These features include writing fluency score, pressure stability index, and the duration of pen pause before and after writing the character. Micro-features at all time points within the same character are statistically aggregated, and statistics are calculated and combined into a character-level mesoscopic feature vector. The character-level mesoscopic features of all characters are globally standardized. The standardized feature vectors are input into a pre-trained machine learning classifier, which outputs the probability distribution of each character belonging to a metacognitive state category. The category with the highest probability is selected as the metacognitive state label for that character, and the confidence score of that label is also output. All labels and confidence scores are then arranged in the original text order to obtain a labeled character sequence.

[0058] Among them, statistical aggregation refers to summarizing and calculating a batch of data, extracting scattered raw data into statistical results that can reflect the overall characteristics; global standardization refers to unifying the rules, caliber, units, and formats of all data / indicators, eliminating local differences, and achieving overall uniformity and standardization.

[0059] Step S103: Using characters in the labeled character sequence as nodes, construct a directed acyclic causal graph, and based on the writing quality score, perform parameter learning on the directed acyclic causal graph to generate causal effect parameters.

[0060] For example, the overall writing quality score of the essay is obtained in two ways: first, by averaging the scores independently given by at least two language arts teachers according to a standard essay scoring rubric; second, by automatically generating the score using a pre-trained document-level Transformer model. A quality contribution weight is calculated for each character. This weight is obtained by normalizing the gradient of the writing quality score output by the document-level Transformer model to the character's input embedding vector, reflecting the degree to which the character's handwriting quality contributes to the final score. Each character in the labeled character sequence is treated as a node, and sentence nodes and discourse nodes are added to generate a three-layer directed acyclic graph: the bottom layer consists of character nodes, each carrying attributes including: metacognitive state label, standardized character-level feature vector, semantic role identifier, and stroke complexity; the middle layer consists of sentence nodes, each sentence node acting as a latent variable connecting all character nodes contained in that sentence; the top layer consists of discourse nodes, connecting all sentence nodes. We further define two types of causal edges: the first type points from the metacognitive state label of a character node to the sentence-level quality index of the corresponding sentence node, which is used to model how cognitive difficulties at the character level affect quality dimensions such as sentence coherence and grammatical correctness; the second type points directly from a character node to the quality contribution weight of that character.

[0061] Normalization refers to mapping data of different magnitudes and units to a fixed interval to eliminate the influence of differences in units and numerical values. The stroke complexity index can be obtained from a Chinese character structure database. The character input embedding vector is obtained by: extracting the text content of each character from a labeled character sequence and mapping it to a one-hot encoding vector or index using a pre-trained vocabulary; converting this index into a dense low-dimensional semantic vector using a trainable embedding matrix, and simultaneously generating a positional encoding vector for the character, where each dimension can be calculated using a cosine function based on the character's absolute position in the sequence; and then adding the semantic vector and positional encoding vector element-wise to obtain the character input embedding vector. A Chinese character structure database is a database system specifically designed for storing, organizing, and managing the structural information of Chinese characters. In this database, the stroke complexity index is a quantitative value of the visual and writing difficulty of each Chinese character, used to measure the complexity of its form. The pre-trained vocabulary is similar to a dictionary. It pre-stores all commonly used characters / words, each with a unique number; semantic role identifiers refer to subjects, predicates, objects, etc., which can be obtained by analyzing the text content of each character in a labeled character sequence using dependency syntax; dependency syntax is a linguistic theory and method used to analyze the grammatical relationships between words in a sentence. Its core idea is that each word in a sentence depends on another word, forming a directed, tree-like dependency structure; pre-trained document-level Transformer models refer to deep learning models based on the Transformer architecture that are pre-trained on massive amounts of general text data and can encode entire documents as input. These models first undergo a first-stage learning process on extremely large and widely covered general corpora. Through language modeling tasks, the models acquire general language representation capabilities. Based on this, they are applied to the specific task of automatic essay scoring, and fine-tuned using a large number of essay samples with human scoring, thereby learning how to effectively score essays.

[0062] For example, structural equation modeling can be used to learn the parameters of a directed acyclic causal graph: for each character node in the causal graph Establish a causal equation: By simultaneously solving the equations for all character nodes in the causal graph, a joint likelihood function is constructed for the observed data. This function represents the joint likelihood function of each character's metacognitive features, sentence quality observations, and writing quality scores. Since the model contains latent variables (i.e., no sentence quality observations exist), the coefficients of all causal paths can be iteratively optimized using gradient descent by maximizing the log-likelihood function of the observed data. Iteration stops when the parameter changes are less than a preset threshold. The estimated set of coefficients for each causal path is then used as the causal effect parameters of metacognitive features on the writing quality scores.

[0063] in, This represents the potential quality value of a character node, which is approximated in actual observations using the quality contribution weight. Let be the set of all parent node variables pointing to node j in the causal graph; This is the metacognitive feature vector of the character; This is a nonlinear function implemented using a multilayer perceptron, used to fit the complex mapping relationship between the parent node and the current node; The random error term follows a normal distribution with a mean of zero. Parameter learning refers to the process by which a model learns task capabilities by continuously adjusting its parameters in the data to fit data patterns and reduce errors. Structural equation modeling is a statistical modeling method whose core idea is to formally transform the causal or hypothetical relationships between variables expressed by a directed acyclic graph into a set of simultaneous mathematical equations. A multilayer perceptron is the most basic fully connected neural network, composed of multiple layers of neurons, capable of solving nonlinear problems. Gradient descent is an iterative algorithm used to optimize an objective function. It calculates the gradient of the objective function with respect to the model parameters (i.e., the first derivative) and updates the parameters in the opposite direction of the gradient, thereby gradually approaching the local or global minimum of the function, depending on the properties of the function. A preset threshold is a pre-set critical value / judgment criterion used to determine whether the iterative calculation process has reached convergence.

[0064] Step S104: Based on the causal effect parameters, counterfactual reasoning is performed on each character to generate the causal effect strength. Based on the causal effect strength, reverse path tracing is performed in the directed acyclic causal graph to obtain the attribution path chain.

[0065] Counterfactual reasoning refers to a causal inference method that infers the outcome state of one or more variables under assumed conditions without changing the actual causal structure. Reverse path tracing refers to an analytical method that, in a causal graph, starts from a target node and traverses along the reverse direction of the causal edges to identify and record all upstream feature nodes and their influence paths that have a causal impact on the target node.

[0066] For example, a counterfactual causal graph with the exact same structure as the original causal graph is created, replicating the attributes of all nodes and their parent-child relationships. In the counterfactual graph, the metacognitive state label of the current character node is replaced with a pre-defined baseline state, such as replacing vocabulary extraction difficulty with fluency. The corresponding metacognitive feature vector components are taken as typical values ​​of the character in the fluent writing state in historical data. Keeping all other node attributes unchanged, the quality contribution weights of each node in the counterfactual graph are recalculated using the learned structural equation model parameters. The difference between the original quality contribution weights and the counterfactual quality contribution weights is calculated, and this difference is defined as the causal effect strength of the character's current metacognitive state on the writing quality score. A positive strength indicates that the character's current metacognitive state contributes positively to the score relative to the fluency state, while a negative strength indicates a negative impact. The absolute value represents the degree of influence. Reverse path tracing based on causal effect strength in the causal graph: Starting from a character node, a depth-first traversal is performed along the reverse edges pointing to the quality contribution weight of that node in the directed acyclic causal graph. During the traversal, each traversed feature node and its causal direction with the current node (positive or negative correlation) are recorded, along with the effect size extracted from the structural equation model, i.e., the partial regression coefficient of that feature node on downstream nodes. This tracing continues until a leaf node or a document node is reached. All nodes and edges traversed in sequence are connected to form an ordered attribution path chain. This chain starts with a specific measurable metacognitive feature, ends with a writing quality score, and includes intermediate variables such as character-level quality contribution weights and sentence-level quality indicators.

[0067] Among them, leaf nodes refer to metacognitive features without any parent nodes, such as original measurements like the duration of the pause before writing and the variance of pressure; depth-first traversal refers to prioritizing exploration along a branch as far down / forward as possible, and after reaching the end, backtracking to the previous level node, and then traversing other unvisited branches; partial regression coefficients refer to the parameter vectors that make the log-likelihood function reach its maximum value by using the gradient descent method.

[0068] Step S105: Generate a Chinese language tutoring report based on metacognitive state labels, causal effect strength, and attribution path chain.

[0069] For example, the metacognitive state labels, causal effect strengths, and attribution path chains of all characters are aggregated into a structured dataset. A tutoring report for Chinese language teaching is generated based on this dataset, which may include three levels of content. The first level is a timeline view, where the metacognitive state labels of each character are marked with different colors at their corresponding text positions according to their original order in the essay, and a bar chart of the causal effect strength of that character is drawn below each position. The second level is diagnostic explanations, converting each attribution path chain into natural language descriptions. For example, a syntactic organization problem at the character "but," such as a pause after writing exceeding twice the global average, leads to a decrease in the quality contribution weight of that character, thus lowering the sentence coherence score. The third level is targeted suggestions, retrieving corresponding practice content and strategy hints from a pre-set teaching suggestion knowledge base based on frequently occurring metacognitive state label types, and associating these suggestions with specific character positions. The above content is packaged into an interactive HTML document and delivered to teachers and students via a cloud platform.

[0070] Among them, the pre-built teaching suggestion knowledge base refers to a pre-constructed knowledge base specifically designed to provide teaching suggestions. Based on teaching needs, it can quickly retrieve and match corresponding practice exercises, while simultaneously accessing accompanying teaching methods, guidance techniques, and other strategic prompts to assist in the implementation of teaching. HTML (HyperText Markup Language) is a markup language used to build web page structures; it is not a programming language. It is used to define the semantics and structure of web page content, allowing browsers to know how to display elements such as text and images. The cloud platform refers to a shared service platform that centrally integrates computing power, storage, software, and network resources based on the internet. The teacher's end refers to the user interface or functional module in educational software designed and used specifically for teachers. The student's end refers to the client application or interface in educational software used by student users.

[0071] This embodiment provides a cloud-based Chinese language teaching aid method that semantically binds temporal handwriting data during the writing process to the final text through cross-modal spatiotemporal alignment, enabling each character to carry a dynamic fingerprint of the writing process. Based on point-level and character-level features, metacognitive labels reflecting students' cognitive states such as vocabulary extraction and syntactic organization are extracted. A causal graph model is used to quantify the causal effect of each character's metacognitive features on writing quality. Counterfactual reasoning is used to trace the specific attribution path, revealing how cognitive impairment affects the final score. Finally, an interpretable, evidence-chain-supported personalized tutoring report is generated.

[0072] It can effectively solve the problem that existing technologies cannot detect the implicit cognitive activities of students in the writing process. By providing transparent attribution explanations through counterfactual reasoning, it enables teachers to accurately locate the difficulties students encounter in specific cognitive aspects such as vocabulary retrieval, syntax organization, or text conception. At the same time, it provides students with evidence-based personalized learning suggestions, significantly improving the diagnostic and teaching value of the Chinese language teaching assistance system.

[0073] In one embodiment, such as Figure 2 As shown, dynamic handwriting data during the writing process and static text data after completion are obtained. Handwriting-text cross-modal spatiotemporal alignment and semantic binding are performed on the static text data and dynamic handwriting data to generate an enhanced document data structure, including:

[0074] Step S201: Obtain dynamic handwriting data of the student during the writing process and the final draft page image after the writing is completed; wherein, the dynamic handwriting data includes time sequence, coordinates, pressure and pen posture.

[0075] For example, a digital writing tablet with electromagnetic induction can be used to collect dynamic handwriting data generated by students during the writing process in real time. This writing tablet can continuously capture the trajectory of the pen tip on the tablet surface at a fixed time sampling rate. For each sampling moment, the following multi-dimensional physical quantities are recorded: the horizontal and vertical coordinates of the pen tip in the two-dimensional coordinate system of the tablet surface; the pressure value applied by the pen tip to the tablet surface, quantized and output through a pressure sensor; the tilt angle of the pen body relative to the normal of the tablet surface and the azimuth angle of the pen tip, used to describe the pen's posture; each sampling point is accompanied by a millisecond-level timestamp provided by the system clock. All sampling points are organized chronologically into an original handwriting sampling point sequence. When it is detected that the student has completed writing the entire essay, a rendered image of all ink marks on the current writing tablet is acquired, i.e., the final page image. This image is in bitmap format with a resolution of no less than the required number of dots per inch (dpi) to ensure sufficient accuracy for subsequent optical character recognition.

[0076] Among them, pressure sensor refers to a device that converts mechanical pressure / force into electrical signals; system clock refers to the core hardware component in a computer used for timing and synchronization, responsible for maintaining the current time and date, and providing a time reference for process scheduling, log recording, etc.; dot-per-inch requirement refers to the standard or minimum specification set for the number of pixels that a device or document can display within one inch of its length in the fields of image processing, printing, or display.

[0077] Step S202: Extract derived features from the dynamic handwriting data to obtain derived features, and fuse the dynamic handwriting data and derived features to obtain enhanced handwriting time series data; wherein, the derived features include instantaneous velocity, instantaneous acceleration, pressure change gradient and pen pause time.

[0078] Among them, derived feature extraction refers to using existing original features to generate new features that are more valuable and more suitable for the model through calculation, combination and transformation. The core is to create new data from old data.

[0079] For example, derived features are extracted from dynamic handwriting data to obtain derived features such as instantaneous velocity, instantaneous acceleration, pressure change gradient, and pen pause time. The dynamic handwriting data and derived features are then fused to obtain enhanced handwriting time series data.

[0080] Step S203: Perform text recognition on the final draft page image to generate static text data.

[0081] Text recognition refers to the technology that allows machines to automatically extract and recognize text content from images such as pictures, screenshots, and scanned documents.

[0082] For example, the OCR engine deployed on the cloud platform is invoked to perform character detection and recognition on the final page image. The output includes the text content of each character and the coordinates of its smallest bounding rectangle in the image's pixel coordinate system (i.e., the pixel positions of the top-left, top-right, bottom-right, and bottom-left points), as well as the line number and its index within the line. All characters are arranged into a linear sequence according to their natural reading order from left to right and top to bottom in the image, while preserving the spatial position information of each character. This sequence constitutes the static text data. The image pixel coordinate system is the most basic two-dimensional Cartesian coordinate system for images, describing the position of each pixel within the image.

[0083] Step S204: Perform handwriting-text cross-modal spatiotemporal alignment and semantic binding on static text data and enhanced handwriting time-series data to generate an enhanced document data structure.

[0084] Among them, handwriting-text cross-modal spatiotemporal alignment and semantic binding refers to the technical process of jointly aligning dynamic handwriting time-series data and static text data in two dimensions, namely time axis and spatial coordinate system, and binding them to a unified semantic space.

[0085] For example, handwriting-text cross-modal spatiotemporal alignment and semantic binding are performed on static text data and enhanced handwriting time-series data to generate an enhanced document data structure.

[0086] In one embodiment, handwriting-text cross-modal spatiotemporal alignment and semantic binding are performed on static text data and enhanced handwriting temporal data to generate an enhanced document data structure, including:

[0087] Step S301: Based on the coordinates of the center point of the minimum bounding rectangle of at least three characters in the static text data and the physical coordinates of the corresponding handwriting points in the enhanced handwriting time series data, the affine transformation matrix is ​​obtained by fitting using the least squares method.

[0088] For example, at least three spatially evenly distributed characters are selected from static text data, and the coordinates of the center point of the smallest bounding rectangle of each character in the image pixel coordinate system are obtained. , and The physical coordinates of the writing points corresponding to these characters are retrieved from the enhanced handwriting time series data. This backtracking is achieved by finding the handwriting sampling point that best matches the spatial location of the character's minimum bounding rectangle. It assumes an affine transformation relationship between the two coordinate systems: for any point, its image coordinates... and physical coordinates satisfy The above Substituting the control points, an overdetermined linear equation system is constructed. Due to noise in actual measurements, the affine transformation matrix can be solved using the least squares method. That is, minimizing the sum of squared reprojection errors of all control points. By solving this optimization problem, we obtain... Numerical solutions for the six independent parameters.

[0089] in, For parameters that include rotation and scaling components, The translation component is used; the least squares method is a mathematical optimization method used to fit data. It finds an optimal curve / straight line that fits the discrete observation data, minimizing the sum of squared errors of all observations and fitted values, rather than minimizing individual errors.

[0090] Step S302: Based on the affine transformation matrix, the physical coordinates of all handwriting sampling points in the enhanced handwriting time series data are transformed to the image pixel coordinate system of the static text data to obtain the aligned handwriting sampling point sequence.

[0091] For example, the affine transformation matrix Applied to each handwriting sampling point in the enhanced handwriting time-series data. For sampling points in the physical coordinate system. Construct its homogeneous coordinate representation Then multiply by the affine transformation matrix on the left. : This gives the corresponding coordinates of the point in the image pixel coordinate system. The above transformation is performed on all sampling points in the enhanced handwriting time-series data one by one, while preserving the original timestamp, pressure value, pen posture, and derived features such as instantaneous velocity, instantaneous acceleration, and pressure change gradient of each sampling point. After the transformation, the spatial positions of all handwriting sampling points are now in the same image coordinate system as the characters in the static text data. These spatially aligned sampling points are then rearranged in ascending order of timestamps to generate an aligned handwriting sampling point sequence. Each point in this sequence includes image coordinates, timestamp, pressure, and dynamic features.

[0092] In physics, a physical coordinate system is a spatial reference frame used to describe the position, motion, and interaction of objects. It typically consists of an origin and a set of orthogonal or non-orthogonal coordinate axes, used to quantitatively determine the position of any point in space.

[0093] Step S303: The dynamic time warping algorithm is used to perform coarse alignment processing on the aligned handwriting sampling point sequence and the character sequence in the static text data to obtain the handwriting sampling point index range corresponding to each character.

[0094] Dynamic time warping is an algorithm used to measure the similarity between two time series. Based on the idea of ​​dynamic programming, it constructs a distance matrix and calculates the local distance of each point in the two series. It then finds a warping path with the minimum cumulative distance in the matrix, allowing local time axis scaling and repeating / skipping points. The minimum cumulative distance is the similarity between the two series, and the smaller the value, the more similar they are.

[0095] For example, the sequence of aligned handwriting sampling points is represented as a sequence. Each of them Carrying image coordinates and timestamp Representing character sequences in static text data as sequences Each of them Carry the coordinates of the region containing its smallest bounding rectangle. Construct a Cumulative cost matrix The matrix is ​​then filled using a dynamic time warping algorithm. For each element in the matrix... Its recursive formula is: After filling in the complete cumulative cost matrix, from Start backtracking along the minimum cost path to The backtracking path indicates each character The corresponding continuous sampling point index range ,in and These represent the first and last sample points of the character in the sequence. The position within the range. The set of sample point indexes for all characters is used as the coarse alignment result.

[0096] in, This represents the total cost of aligning the i-th stroke point and the j-th character to this point; To skip the penalty; Insertion penalty; skip penalty and insertion penalty are used to allow for misalignment of some elements in the sequence, accommodating insertions or omissions during the writing process; Alignment cost is defined as the difference between spatial distance and dynamic characteristics. Weighted sum: ; and These are pre-defined weighting coefficients used to adjust the importance of the space / dynamics. For spatial distance, sampling points can be used. to character The Euclidean distance of the minimum bounding rectangle; To account for differences in dynamic characteristics, the instantaneous velocity and pressure values ​​carried in the sampling points can be used to reflect the consistency of writing rhythm.

[0097] Step S304: Normalize the duration and number of aligned handwriting sampling points within the index interval of each handwriting sampling point to obtain normalized time points.

[0098] For example, for each character The corresponding original sampling point index range Extract the aligned handwriting sampling point subsequence within this interval, and denot the number of sampling points contained therein as... The actual writing time for the entire interval is recorded as follows: Because the writing time and number of sampling points vary significantly among different characters, these sequences are normalized so that each character is represented as a sequence of a fixed number of time points: a pre-defined number of normalized time points is set. ,For example This value can be selected empirically; it needs to be large enough to preserve the details of the writing dynamics but not so large as to cause computational redundancy. (The interval is...) Evenly divided into Each time segment, each normalized time point The corresponding time is: For each Determine the insertion position in the original sampling point sequence, i.e., find the index. Make Then, based on the data from adjacent original sampling points, physical quantities such as coordinates and pressure at the normalized time point can be calculated using interpolation methods. After the above processing, each character... Generated Each time point is a normalized time point, and each time point carries the interpolated image coordinates. ,pressure It is equivalent to the instantaneous velocity derived from the dynamic parameters.

[0099] Interpolation refers to a method that, given a set of discrete data points, constructs a continuous curve / function that passes strictly through all known points, thereby estimating the values ​​of unknown points within an interval.

[0100] Step S305: Extract writing dynamics parameters at each normalized time point and summarize the writing dynamics parameters to obtain the writing dynamics parameter sequence of the corresponding character; wherein, the writing dynamics parameters include instantaneous velocity, pressure, acceleration and pressure gradient value.

[0101] For example, based on each character of Normalization time point and its corresponding interpolated image coordinates Pressure value For each character, iterate through all normalized time points, based on the displacement difference between adjacent points. , and time difference First calculate the magnitude of the instantaneous velocity. and direction angle Then, acceleration is obtained from the rate of change of velocity. At the same time, the pressure gradient can be obtained based on the pressure change. and the original pressure value Included together. For the start and end points of the sequence, i.e. or Forward or backward differencing can be used instead of central differencing to ensure the integrity of numerical calculations. After calculating at each time point, the resulting five sets of sequences—instantaneous velocity magnitude sequence, instantaneous velocity direction angle sequence, acceleration sequence, pressure sequence, and pressure gradient sequence—are organized in chronological order into a shape of [shape missing]. The tensor is used as the sequence of writing dynamics parameters for the character. Forward difference refers to calculating the derivative using the difference between the current point and the next point; backward difference refers to calculating the derivative using the difference between the current point and the previous point; and central difference refers to calculating the derivative using the difference between the previous point and the next point.

[0102] Step S306: Based on the intersection-union ratio of the bounding box of each stroke in the aligned handwriting sampling point sequence and the minimum bounding rectangle of each character in the static text data, combined with the preset tampering judgment threshold, the tampering behavior is judged, and the judgment result is obtained.

[0103] For example, the aligned handwriting sampling point sequence is grouped according to strokes, with each stroke corresponding to a continuous writing action from pen entry to pen exit, resulting in a stroke set. For each stroke, its minimum bounding rectangle is calculated, which is the smallest axis-aligned rectangle containing all sampling points of that stroke. The minimum bounding rectangle of each character is obtained from the static text data. The intersection-union ratio (IUR) of each stroke-character pair is calculated. If there is a character whose IUR is greater than a preset tampering threshold, the difference between the timestamp of that stroke and the timestamp of the character's initial writing is checked to see if it is less than a preset time window threshold, thus excluding natural overlap of adjacent strokes during normal writing. If the time difference also meets the condition, the stroke is judged as an tampering action, and the set of character indices affected by the tampering is recorded. For each stroke judged as tampering, its dynamic characteristics are extracted: total writing time, average writing speed, pressure fluctuation range, acceleration variance, etc., and the tampering type is identified, such as complete coverage, local addition, etc. The final result includes the number of times each character was altered, the dynamic parameters of each alteration, the type of alteration, and the temporal order of the alterations.

[0104] The formula for calculating the intersection-union ratio is: The numerator is the area of ​​the overlapping region of the two rectangles, and the denominator is the total area covered by the two rectangles. For intersection, union, and comparison, The smallest bounding rectangle of a stroke. The minimum bounding rectangle of the character; the preset time window threshold is a pre-defined continuous time range used to define the time period for statistics, judgment, and effectiveness; the preset tampering judgment threshold is a pre-defined quantitative standard line used to determine whether it belongs to tampering, usually between 0.2 and 0.5.

[0105] Step S307: Integrate the writing dynamics parameter sequence and judgment result of each character into the character structure of the static text data to obtain the enhanced document data structure.

[0106] For example, a new empty data structure is created to store the enhanced document, with static text data as its basic framework. First, each character in the static text data is traversed, and a corresponding character node is assigned to it. For each character node, the sequence of writing dynamics parameters for that character is stored in the node as key-value pairs; the correction judgment result for that character is inserted into the correction history list field of the node; the image coordinate sequence corresponding to the normalized time point of the character and the original sampling point index range are extracted and stored in the node for visualization playback. The original character Unicode values, minimum bounding box coordinates, and line numbers from the static text data are preserved. After filling all character nodes, these character nodes are linked into a sequence according to the order in the original text to generate the enhanced document data structure. This data structure is a hierarchical object. The outermost layer records the overall metadata of the document, such as page width, page height, and total number of characters. The inner layer is an array of character nodes, each containing text semantic information, spatial location information, writing dynamics fingerprint, and the correction history throughout its entire lifecycle.

[0107] In one embodiment, derived features are extracted from dynamic handwriting data to obtain derived features. The dynamic handwriting data and derived features are then fused to obtain enhanced handwriting time-series data, including:

[0108] Step S401: Filter each stroke in the dynamic handwriting data to obtain smooth handwriting sampling points, and arrange all smooth handwriting sampling points in chronological order to obtain a smooth handwriting sampling point sequence.

[0109] For example, the raw dynamic handwriting data is organized by stroke, with each stroke containing several points continuously sampled from the beginning to the end of the stroke. Gaussian-weighted moving average filters are applied to the coordinate sequence and pressure sequence of each stroke. Specifically, for the first stroke in the current stroke... Using a sampling point as the center, take samples from the front and back of that point. The points constitute the total length. A sliding window is used, and the Gaussian weights are calculated using the following formula: The smoothed coordinates are obtained by weighted averaging the coordinates of each point within the window and their corresponding weights. The calculation formula is as follows: Pressure values ​​are processed in the same way. For points at the ends of a stroke that are shorter than the window length, a strategy of symmetrically expanding or reducing the window size can be used. After filtering all strokes, all smoothed sampling points are arranged in global order of their original timestamps to obtain a smoothed handwriting sampling point sequence. Each point includes its smoothed coordinates, pressure, and original timestamp.

[0110] in, This is the index number of the currently processed sampling point within the current stroke. This is the index number of any sampling point within the sliding window, with a value ranging from... arrive ; For the first in the sliding window The Gaussian weighting coefficients of each sampling point are used for weighted averaging; The standard deviation of the Gaussian kernel is used to control the degree of smoothing. The larger the value, the flatter the weight distribution and the stronger the smoothing effect; For the first in the sliding window The original physical coordinates of each sampling point; After Gaussian weighted moving average filtering, the first... Smoothed coordinates of each sampling point; For the first The original timestamp of each sampling point; the strategy of symmetrically expanding or reducing the window size refers to stretching or shrinking the window horizontally and vertically in proportion to the geometric center of the window, while keeping the center point of the window unchanged and the aspect ratio constant.

[0111] Step S402: Based on the coordinate difference and the first time difference between adjacent sampling points in the smooth handwriting sampling point sequence, calculate the instantaneous velocity and instantaneous acceleration of each sampling point, and arrange the instantaneous velocity and instantaneous acceleration in time order to obtain the instantaneous velocity sequence and the instantaneous acceleration sequence.

[0112] For example, a sequence of smoothed handwriting sampling points is obtained, where each sampling point carries smoothed coordinates. Pressure value and original timestamp The total length of the sequence is The time interval between two adjacent sampling points is calculated using the following formula: For indexes From 1 to For each sampling point, calculate the displacement component between it and the next sampling point. , Based on this, according to the formula Find the arc length in two-dimensional space. Based on this arc length and the time interval, calculate the arc length of the second arc. The instantaneous tangential velocity at each sampling point is calculated using the following formula: At the same time, through the arctangent function Calculate the direction angle of the instantaneous velocity. For instantaneous acceleration, it is obtained by calculating the rate of change of adjacent velocities; the formula is: ,in This also represents the time difference between adjacent sampling points, but here it corresponds to the time interval between velocity changes. For the point at the end of the sequence ( Since forward velocity cannot be calculated, the velocity value from the previous moment can be used as filler or discarded directly. All calculated instantaneous velocity and instantaneous acceleration values ​​are arranged in the order of the original sampling points to obtain the instantaneous velocity sequence and instantaneous acceleration sequence.

[0113] Step S403: Calculate the pressure change gradient based on the pressure difference and second time difference between adjacent sampling points in the smooth handwriting sampling point sequence, and arrange the pressure change gradients in time order to obtain the pressure change gradient sequence.

[0114] For example, the smoothing pressure value of each sampling point in the smooth handwriting sampling point sequence is extracted. Similar to step S402, for consecutive sampling points in the sequence, according to the formula... and Calculate the pressure difference and time interval between two adjacent sampling points. Pressure change gradient. Defined as the derivative of pressure with respect to time, it can be approximated using the forward difference method, and the calculation formula is as follows: This gradient reflects the rate of increase or decrease in pressure at the pen tip per unit time; a positive value indicates increasing pressure (pen tip pressing down), and a negative value indicates decreasing pressure (pen tip lifting up). For each valid... Calculate the corresponding pressure change gradient and arrange these values ​​in the original order of the sampling points to obtain the pressure change gradient sequence. For the last sampling point, backward differencing or direct reuse of the previous gradient value can be used.

[0115] Step S404: Based on the smooth handwriting sampling point sequence, extract the pen pause time from the third time difference between the end time of the current stroke and the start time of the next stroke, and arrange the pen pause times in chronological order to obtain the pen pause time sequence.

[0116] For example, if the current stroke is the first... Each stroke, with the timestamp of its last sampling point recorded as... The next stroke, i.e., the first... The timestamp of the first sampling point of each stroke is... Since no new sampling point is generated after the pen tip leaves the board during writing, the difference between these two timestamps represents a pen-suspending pause. A pen-suspending pause indicates that the student is in a state of conceptualization, hesitation, or recalling information during that time, and its duration is positively correlated with cognitive load. By traversing all adjacent stroke pairs, the duration of each pen-suspending pause and its corresponding spatial location are extracted. This can typically be achieved by taking the coordinates of the last sampling point of the current stroke, or by taking the area near the end of the stroke. All extracted pen-suspending pause times are arranged in stroke order to obtain a pen-suspending pause time sequence. Simultaneously, the timestamp of each pause and the corresponding character region are recorded.

[0117] Step S405: The smooth handwriting sampling point sequence, timestamp sequence, instantaneous velocity sequence, instantaneous acceleration sequence, pressure change gradient sequence and pen pause time sequence of each stroke are time-axis aligned and stitched together to obtain the strong handwriting time series data of each sampling point.

[0118] Among them, timeline alignment and splicing refers to aligning multiple segments of material with a unified time reference, first aligning the timestamps, synchronizing the start and end points and rhythm, and then splicing them together in sequence.

[0119] For example, the coordinates of the smooth handwriting sampling points Original timestamp Smoothing pressure Instantaneous velocity and direction angle Instantaneous acceleration Pressure change gradient Press index directly Merge the samples to obtain the basic feature vector for each sampling point. For the pause time, assign it to the sampling points corresponding to the stroke boundaries: Give the first The last sampling point of the first stroke or the first sampling point of the next stroke is used, while the pen pause time field corresponding to the sampling points within the stroke is set to 0 or null. The temporal order of all features is kept strictly consistent, i.e., the first... The feature vector corresponding to each sampling point is ,in Non-zero values ​​are only found at the end points of strokes. After performing the above concatenation on all sampling points, enhanced handwriting temporal data is obtained.

[0120] In one embodiment, the metacognitive features of each character in the enhanced document data structure are mapped to corresponding metacognitive state labels to obtain a labeled character sequence, including:

[0121] Step S501: Based on each character and the sequence of writing dynamics parameters of the character in the enhanced document data structure, calculate the point-level micro-features at each normalized time point; wherein, the point-level micro-features include writing fluency score, pressure stability index, pen suspension pause duration before writing and pen suspension pause duration after writing.

[0122] For example, each character node is traversed from the enhanced document data structure, and each node is bound to a length of... The sequence of writing dynamics parameters was used. For each normalized time point, a writing fluency score and a pressure stability index were calculated. The writing fluency score is the absolute value of the instantaneous rate of change of velocity, i.e. , To smooth the feature speed after processing, a smaller value indicates a smoother and more continuous writing process; the pressure stability index can be characterized by the ratio of the current pressure value to the average pressure value within a local window, specifically... , This is the average pressure within a sliding window centered on the current point. The degree to which this ratio deviates from 1 reflects the severity of force fluctuations. The duration of the pen-holding pause before writing is directly extracted from the enhanced document data structure. and the length of time to pause after writing The former represents the duration of the pause before the character begins writing, and the latter represents the duration of the pause between the completion of one character and the start of the next. These two scalar values ​​are copied to the features of all normalized time points for that character to maintain dimensionality consistency in subsequent aggregation. After the above calculations, each normalized time point for each character obtains a four-dimensional point-level micro-feature vector.

[0123] Step S502: Statistically aggregate the point-level micro-features at all normalized time points within the same character to obtain character-level meso-features; wherein, the character-level meso-features include the mean pressure, mean velocity, mean absolute value of acceleration, pressure variance, velocity variance, velocity coefficient of variation, number of erasures, interquartile range of pressure, and average erasure time.

[0124] For example, for the same character, it The original dynamic parameters and point-level micro-features at each normalized time point are statistically summarized. Specifically, the mean pressure, mean velocity, mean absolute acceleration, pressure variance, velocity variance, velocity coefficient of variation, and pressure interquartile range are calculated. For alteration-related features, the number of alterations and the duration of each alteration are read from the alteration history of the character, and the average alteration duration is calculated (0 if there is no alteration). All statistics are combined to obtain the character-level mesoscopic feature vector. Among them, the velocity coefficient of variation is the ratio of the velocity standard deviation to the velocity mean; the pressure interquartile range is obtained by calculating the difference between the 75th percentile and the 25th percentile after sorting the pressure sequence.

[0125] Step S503: Standardize each character-level mesoscopic feature to obtain standardized character-level features, and summarize the standardized character-level features to obtain a standardized feature vector.

[0126] For example, suppose the entire enhanced document data structure contains a total of Each character, for each feature dimension in the mesoscopic feature vector. Collect all values ​​of all characters in this dimension. According to the formula and Calculate the global mean and global standard deviation for each character. The original values ​​in this dimension are Z-scor standardized. The formula for Z-scor standardization is: .when At this point, the standardized value is set to 0. This standardization is then performed sequentially on all mesoscopic feature dimensions to obtain the character... The standardized character-level feature vectors are generated. Each component of this vector follows a distribution with a mean of 0 and a variance of 1. The standardized feature vectors of all characters are stored in character order to obtain a list of standardized feature vectors.

[0127] Step S504: Input the standardized feature vector into the machine learning classifier and output the metacognitive state label and the confidence score of the metacognitive state label.

[0128] The machine learning classifier is based on the Transformer architecture, comprising an embedding layer and a multi-layer Transformer encoder. During training, this classifier utilizes a large amount of labeled historical writing data, optimizing model parameters with cross-entropy loss. The embedding layer maps the normalized feature vector of each character to a high-dimensional latent vector, then superimposes positional encoding to preserve text order information. Each layer of the multi-layer Transformer encoder contains a multi-head self-attention mechanism and a feedforward neural network. The self-attention mechanism allows the model to capture dependencies between different characters; for example, hesitation in one character may affect the writing performance of the next. The output is transformed by a linear transformation and the Softmax function to obtain the probability distribution of each character belonging to a preset metacognitive state category. The preset metacognitive state categories include fluency, difficulty in word extraction, and syntactic organization problems. A feedforward neural network is an artificial neural network in which information flows unidirectionally from the input layer through one or more hidden layers to the output layer. There are no feedback connections or loop structures between the layers. A linear transformation refers to a linear mapping implemented by matrix multiplication and bias. The Softmax function is an activation function commonly used in machine learning to compress and map any real number vector to [0,1], and the sum of all outputs is 1, which is converted into a probability distribution.

[0129] For example, a standardized feature vector is input into a machine learning classifier, which outputs a probability distribution of each character belonging to a preset metacognitive state category. The category with the highest probability is selected as the metacognitive state label for that character, and the corresponding probability value, or the maximum value, is used as the confidence score.

[0130] Step S505: Arrange the metacognitive state label and confidence score of each character in the order of the original text to obtain a labeled character sequence.

[0131] For example, the metacognitive state label and confidence score of each character are arranged in character order to obtain a metacognitive state label sequence and a confidence score sequence. The metacognitive state label sequence and confidence score sequence are then aligned and encapsulated according to the order in which the characters appear in the original static text, i.e., the natural reading order from the first character to the last character of the document. For each character, a label node is created, which includes three fields: the Unicode text content of the character itself, the metacognitive state label (an enumeration or string value), and the confidence score. These label nodes are then concatenated into a linear sequence, i.e., a labeled character sequence. The length of this sequence is equal to the total number of characters in the static text data and corresponds one-to-one with the character nodes in the enhanced document data structure. This labeled character sequence is then persistently stored on a cloud platform. Persistent storage refers to a storage method where data is not lost after power failure or program exit, and can be preserved long-term; encapsulation refers to binding data (attributes) and methods for manipulating the data together to form a class / object.

[0132] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0133] Based on the same inventive concept, this application also provides a cloud-based Chinese language teaching system for implementing the aforementioned cloud-based Chinese language teaching method. The solution provided by this system is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of a cloud-based Chinese language teaching system provided below can be found in the limitations of the cloud-based Chinese language teaching method described above, and will not be repeated here.

[0134] In one exemplary embodiment, such as Figure 3 As shown, a cloud-based Chinese language teaching aid system 300 is provided, including:

[0135] The data acquisition module 301 is used to acquire dynamic handwriting data of students during the writing process and static text data after the writing is completed, and to perform handwriting-text cross-modal spatiotemporal alignment and semantic binding on the static text data and dynamic handwriting data to generate an enhanced document data structure.

[0136] The tag generation module 302 is used to map the metacognitive features of each character in the enhanced document data structure to the corresponding metacognitive state label, so as to obtain a labeled character sequence; wherein, the metacognitive features include point-level micro features and character-level meso features;

[0137] The parameter learning module 303 is used to construct a directed acyclic causal graph using characters in a labeled character sequence as nodes, and to perform parameter learning on the directed acyclic causal graph based on writing quality scores to generate causal effect parameters.

[0138] The parameter analysis module 304 is used to perform counterfactual reasoning on each character based on the causal effect parameters, generate the causal effect strength, and perform reverse path tracing in the directed acyclic causal graph based on the causal effect strength to obtain the attribution path chain.

[0139] The report generation module 305 is used to generate Chinese language tutoring reports based on metacognitive state labels, causal effect strength, and attribution path chains.

[0140] Furthermore, the data acquisition module 301 is also used for:

[0141] Acquire dynamic handwriting data of students during the writing process and images of the final draft pages after the writing is completed; the dynamic handwriting data includes time sequence, coordinates, pressure, and pen posture;

[0142] Derived features are extracted from dynamic handwriting data to obtain derived features, and the dynamic handwriting data and derived features are fused to obtain enhanced handwriting time series data; among them, the derived features include instantaneous velocity, instantaneous acceleration, pressure change gradient and pen pause time;

[0143] Perform text recognition on the final draft page image to generate static text data;

[0144] We perform cross-modal spatiotemporal alignment and semantic binding of handwriting-text data on static text data and enhanced handwriting time-series data to generate enhanced document data structures.

[0145] Furthermore, the data acquisition module 301 is also used for:

[0146] Based on the coordinates of the center point of the minimum bounding rectangle of at least three characters in the static text data and the physical coordinates of the corresponding handwriting points in the enhanced handwriting time series data, the affine transformation matrix is ​​obtained by fitting the least squares method.

[0147] Based on the affine transformation matrix, the physical coordinates of all handwriting sampling points in the enhanced handwriting time series data are transformed to the image pixel coordinate system of the static text data to obtain the aligned handwriting sampling point sequence.

[0148] A dynamic time warping algorithm is used to coarsely align the handwriting sampling point sequence and the character sequence in the static text data to obtain the handwriting sampling point index range corresponding to each character.

[0149] The time and number of aligned handwriting sampling points within the index interval of each handwriting sampling point are normalized to obtain normalized time points.

[0150] Writing dynamics parameters are extracted at each normalized time point and summarized to obtain the writing dynamics parameter sequence for the corresponding character; among which, the writing dynamics parameters include instantaneous velocity, pressure, acceleration and pressure gradient value;

[0151] Based on the intersection-union ratio of the bounding box of each stroke in the aligned handwriting sampling point sequence and the minimum bounding rectangle of each character in the static text data, combined with the preset tampering judgment threshold, the tampering behavior is judged and the judgment result is obtained.

[0152] The sequence of writing dynamics parameters and the judgment results for each character are integrated into the character structure of the static text data to obtain an enhanced document data structure.

[0153] Furthermore, the data acquisition module 301 is also used for:

[0154] Each stroke in the dynamic handwriting data is filtered to obtain smooth handwriting sampling points. All smooth handwriting sampling points are arranged in chronological order to obtain a smooth handwriting sampling point sequence.

[0155] Based on the coordinate difference and first time difference between adjacent sampling points in the smooth handwriting sampling point sequence, the instantaneous velocity and instantaneous acceleration of each sampling point are calculated, and the instantaneous velocity and instantaneous acceleration are arranged in time order to obtain the instantaneous velocity sequence and instantaneous acceleration sequence respectively;

[0156] Based on the pressure difference and second time difference between adjacent sampling points in the smooth handwriting sampling point sequence, the pressure change gradient is calculated, and the pressure change gradient is arranged in time order to obtain the pressure change gradient sequence.

[0157] Based on the smooth handwriting sampling point sequence, the pen pause time is extracted from the third time difference between the end time of the current stroke and the start time of the next stroke, and the pen pause time is arranged in chronological order to obtain the pen pause time sequence.

[0158] The time axis is aligned and stitched together for the smooth handwriting sampling point sequence, timestamp sequence, instantaneous velocity sequence, instantaneous acceleration sequence, pressure change gradient sequence, and pen pause time sequence for each stroke to obtain the strong handwriting time series data for each sampling point.

[0159] Furthermore, the label generation module 302 is also used for:

[0160] Based on each character and the sequence of writing dynamics parameters in the enhanced document data structure, point-level micro-features are calculated at each normalized time point; among which, point-level micro-features include writing fluency score, pressure stability index, pen-suspending pause duration before writing, and pen-suspending pause duration after writing.

[0161] Statistical aggregation of point-level micro-features at all normalized time points within the same character yields character-level meso-features; among which, character-level meso-features include mean pressure, mean velocity, mean absolute value of acceleration, pressure variance, velocity variance, velocity coefficient of variation, number of erasures, interquartile range of pressure, and average erasure time.

[0162] Each character-level mesoscopic feature is standardized to obtain standardized character-level features, and the standardized character-level features are then aggregated to obtain a standardized feature vector.

[0163] The standardized feature vector is input into the machine learning classifier, which outputs metacognitive state labels and confidence scores of the metacognitive state labels.

[0164] The metacognitive state label and confidence score of each character are arranged in the order of the original text to obtain a labeled character sequence.

[0165] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of a cloud-based language teaching aid method as described above.

[0166] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0167] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0168] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A cloud-based method for assisting Chinese language teaching, characterized in that: The method includes: The system acquires dynamic handwriting data of students during the writing process and static text data after the writing is completed, and performs handwriting-text cross-modal spatiotemporal alignment and semantic binding on the static text data and the dynamic handwriting data to generate an enhanced document data structure. The metacognitive features of each character in the enhanced document data structure are mapped to corresponding metacognitive state labels to obtain a labeled character sequence; wherein, the metacognitive features include point-level micro features and character-level meso features; the character is the smallest semantic unit in the static text data; Using the characters in the labeled character sequence as nodes, a directed acyclic causal graph is constructed, and based on the writing quality score, the directed acyclic causal graph is subjected to parameter learning to generate causal effect parameters. Based on the causal effect parameters, counterfactual reasoning is performed on each character to generate the causal effect strength, and based on the causal effect strength, reverse path tracing is performed in the directed acyclic causal graph to obtain the attribution path chain; A Chinese language tutoring report is generated based on the metacognitive state labels, the causal effect strength, and the attribution path chain.

2. The method according to claim 1, characterized in that, The process of acquiring dynamic handwriting data during the writing process and static text data after completion, and performing handwriting-text cross-modal spatiotemporal alignment and semantic binding on the static text data and the dynamic handwriting data to generate an enhanced document data structure includes: The system acquires dynamic handwriting data of students during the writing process and images of the final draft pages after the writing is completed; wherein, the dynamic handwriting data includes time sequence, coordinates, pressure, and pen posture; Derived features are extracted from the dynamic handwriting data to obtain derived features, and the dynamic handwriting data and the derived features are fused to obtain enhanced handwriting temporal data; wherein, the derived features include instantaneous velocity, instantaneous acceleration, pressure change gradient and pen pause time; Perform text recognition on the final draft page image to generate static text data; The static text data and the enhanced handwriting time-series data are subjected to handwriting-text cross-modal spatiotemporal alignment and semantic binding to generate the enhanced document data structure.

3. The method according to claim 2, characterized in that, The step of performing handwriting-text cross-modal spatiotemporal alignment and semantic binding on the static text data and the enhanced handwriting temporal data to generate an enhanced document data structure includes: Based on the coordinates of the center point of the minimum bounding rectangle of at least three characters in the static text data and the physical coordinates of the corresponding handwriting point in the enhanced handwriting time series data, the affine transformation matrix is ​​obtained by fitting the least squares method. Based on the affine transformation matrix, the physical coordinates of all handwriting sampling points in the enhanced handwriting time series data are transformed into the image pixel coordinate system of the static text data to obtain an aligned handwriting sampling point sequence. A dynamic time warping algorithm is used to coarsely align the aligned handwriting sampling point sequence and the character sequence in the static text data to obtain the handwriting sampling point index range corresponding to each character. The time and number of points of the aligned handwriting sampling point sequence within each index interval of the handwriting sampling point are normalized to obtain the normalized time point; At each normalized time point, writing dynamics parameters are extracted and summarized to obtain a sequence of writing dynamics parameters for the corresponding character; wherein, the writing dynamics parameters include instantaneous velocity, pressure, acceleration, and pressure gradient value; Based on the intersection-union ratio of the bounding box of each stroke in the aligned handwriting sampling point sequence and the minimum bounding rectangle of each character in the static text data, combined with the preset tampering judgment threshold, the tampering behavior is judged and the judgment result is obtained. The sequence of writing dynamics parameters for each character and the judgment result are integrated into the character structure of the static text data to obtain the enhanced document data structure.

4. The method according to claim 2, characterized in that, The process of extracting derived features from the dynamic handwriting data to obtain derived features, and then fusing the dynamic handwriting data and the derived features to obtain enhanced handwriting time-series data includes: Each stroke in the dynamic handwriting data is filtered to obtain smooth handwriting sampling points. All the smooth handwriting sampling points are arranged in chronological order to obtain a smooth handwriting sampling point sequence. Based on the coordinate difference and the first time difference between adjacent sampling points in the smooth handwriting sampling point sequence, the instantaneous velocity and instantaneous acceleration of each sampling point are calculated, and the instantaneous velocity and instantaneous acceleration are arranged in chronological order to obtain the instantaneous velocity sequence and the instantaneous acceleration sequence. Based on the pressure difference and the second time difference between adjacent sampling points in the smooth handwriting sampling point sequence, the pressure change gradient is calculated, and the pressure change gradient is arranged in chronological order to obtain the pressure change gradient sequence. Based on the smooth handwriting sampling point sequence, the pen pause time is extracted from the third time difference between the end time of the current stroke and the start time of the next stroke, and the pen pause time is arranged in chronological order to obtain the pen pause time sequence. The time axis is aligned and stitched together for the smooth handwriting sampling point sequence, timestamp sequence, instantaneous velocity sequence, instantaneous acceleration sequence, pressure change gradient sequence, and pen pause time sequence for each stroke to obtain the strong handwriting time series data for each sampling point.

5. The method according to claim 1, characterized in that, The step of mapping the metacognitive features of each character in the enhanced document data structure to a corresponding metacognitive state label to obtain a labeled character sequence includes: Based on each character in the enhanced document data structure and the sequence of writing dynamics parameters of the character, point-level micro-features at each normalized time point are calculated; wherein, the point-level micro-features include writing fluency score, pressure stability index, pen-suspending pause duration before writing, and pen-suspending pause duration after writing; Statistical aggregation is performed on the point-level micro-features at all normalized time points within the same character to obtain character-level meso-features; wherein, the character-level meso-features include mean pressure, mean velocity, mean absolute value of acceleration, pressure variance, velocity variance, velocity coefficient of variation, number of erasures, interquartile range of pressure, and average erasure time. Each of the character-level mesoscopic features is standardized to obtain standardized character-level features, and the standardized character-level features are then aggregated to obtain a standardized feature vector. The standardized feature vector is input into a machine learning classifier, which outputs metacognitive state labels and confidence scores of the metacognitive state labels. The metacognitive state label and confidence score of each character are arranged in the original text order to obtain the labeled character sequence.

6. A cloud-based Chinese language teaching aid system, characterized in that, The system includes: The data acquisition module is used to acquire dynamic handwriting data of students during the writing process and static text data after the writing is completed, and to perform handwriting-text cross-modal spatiotemporal alignment and semantic binding on the static text data and the dynamic handwriting data to generate an enhanced document data structure. The tag generation module is used to map the metacognitive features of each character in the enhanced document data structure to the corresponding metacognitive state tags, thereby obtaining a tagged character sequence; wherein, the metacognitive features include point-level micro features and character-level meso features; The parameter learning module is used to construct a directed acyclic causal graph using the characters in the labeled character sequence as nodes, and to perform parameter learning on the directed acyclic causal graph based on the writing quality score to generate causal effect parameters. The parameter analysis module is used to perform counterfactual reasoning on each character based on the causal effect parameters, generate the causal effect strength, and perform reverse path tracing on the directed acyclic causal graph based on the causal effect strength to obtain the attribution path chain. The report generation module is used to generate a Chinese language tutoring report based on the metacognitive state labels, the causal effect strength, and the attribution path chain.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images