Multi-modal learning data acquisition and standardization method based on smart writing system

Abstract

The invention provides a multi-modal learning data acquisition and standardization method based on a smart writing system, and relates to the technical field of data processing. Disturbance signals are acquired through multi-modal sequences such as pressure, speed, stroke continuity, sight line coordinates and voice energy; nodes are generated in a node type mapping table according to the disturbance amplitude interval number, the change rate interval number and the duration interval number; calculating a disturbance amplitude difference, a change rate difference and a node time interval in a preset time window to form an edge parameter; compressing the node set and the edge set through the node difference degree and the edge parameter difference degree; and performing direction verification and parameter truncation on the structure according to the topology constraint table, and finally outputting a standardized causal graph structure comprising a node list area, an edge list area and a topology description area. According to the method, output structure consistency can be kept under the condition that the number of multiple modes is inconsistent or partial modes are missing, and the method is suitable for scenes such as learning behavior collection, writing process modeling and educational data processing.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method for multimodal learning data acquisition and standardization based on an intelligent writing system. Background Technology

[0002] With the popularization of digital teaching and intelligent educational devices, data acquisition systems based on handwriting behavior are increasingly being applied to scenarios such as classroom learning, online assignments, remote teaching, and learning behavior analysis. Existing intelligent handwriting systems are typically equipped with multiple sensing modules, including handwriting acquisition, pressure sensing, eye tracking, and voice acquisition, which can generate a large amount of multimodal raw data containing different physical quantities, sampling frequencies, and data formats during the learning process. This data has certain advantages in reflecting the details of learner behavior, but due to the large differences in sampling frequencies among the multimodal components, and the inconsistencies in the time scale, numerical range, and variation patterns of signals such as pressure, speed, eye contact, and voice, it is difficult to directly process and structure this data in a unified manner.

[0003] Current multimodal data processing methods often employ feature fusion, time synchronization, signal alignment, or deep learning encoding to map multimodal data to a unified feature space. However, these methods typically require separate feature processing for each modality, and different modalities exhibit significant differences in sampling frequency, data gaps, signal noise, and temporal integrity, resulting in technical gaps in structured modeling, standardized data representation, and cross-modal consistent output. Furthermore, existing handwriting behavior modeling methods often rely on handwriting trajectory reconstruction or image-based handwriting recognition processes, enabling structured representation only for a single modality and making it difficult to establish a unified cross-modal structure when multiple modalities are involved.

[0004] Furthermore, in learning behavior analysis scenarios, it is necessary to identify instantaneous perturbation information related to changes in learning state, cognitive changes, and behavioral transitions from multimodal data. However, in existing technologies, the relevant perturbation signals are often directly represented by raw amplitude, raw coordinates, or raw energy sequences, which lack the ability to express them uniformly across modalities. It is difficult to form a structured link with causal order between perturbation points of different modalities. The structure is difficult to maintain consistency when modalities are missing or the number of modalities is inconsistent. The final output data structure often lacks a unified, compressible, and standardized representation method, which is not conducive to the backend system performing general processing on multiple learning tasks, multiple terminal types, and different data conditions. Summary of the Invention

[0005] Technical problems to be solved To address the shortcomings of existing technologies, this invention provides a method for multimodal learning data acquisition and standardization based on an intelligent writing system, thus solving the problems of existing technologies.

[0006] Technical solution To achieve the above objectives, the present invention provides a method for multimodal learning data acquisition and standardization based on an intelligent writing system, comprising the following steps: Sp1: Disturbance signal acquisition: Acquire at least one disturbance signal generated by the intelligent writing system during the learning process. The disturbance signal includes pressure change in the pressure sampling sequence, speed change in the handwriting sampling sequence, interruption marker in the stroke continuity sequence, angle change in the line-of-sight coordinate sequence, or short-term amplitude change in the speech amplitude sequence.

[0007] Sp2: Node Generation: Based on the preset disturbance classification rules, the collected disturbance signals are mapped to the node type mapping table according to their amplitude range, rate of change range and duration range, and a node set is generated based on the mapping results.

[0008] Sp3: Edge Establishment: Within a preset time window, edges are established between nodes based on the order of their appearance and the numerical relationship of the corresponding disturbance signals. Edge parameters are calculated, which are obtained by combining the disturbance amplitude difference, the rate of change difference, and the node time interval according to a preset calculation formula.

[0009] Sp4: Structure Compression: Based on preset node merging and edge merging rules, the node set and edge set are compressed to generate a compressed node set and a compressed edge set.

[0010] Sp5: Topology Normalization: Based on the preset topology constraint table, perform topology adjustment on the compressed node set and edge set to make them conform to the requirements of connection direction, node hierarchy and edge parameter range in the topology constraint table.

[0011] Sp6: Structural Output: Outputs a causal graph structure containing a set of nodes, a set of edges, and topological description information, but does not contain numerical data that can be used to recover the original perturbation signal sequence in Sp1.

[0012] Preferably, Sp2 further comprises: Sp2-1: Sort the disturbance signals according to the sampling time; Sp2-2: Calculate the interval number of disturbance amplitude, the interval number of change rate, and the interval number of duration for each disturbance signal; Sp2-3: Look up the corresponding row and column index in the node type mapping table based on the three numbers; Sp2-4: Determine the node type based on the index and write the three types of numbers of the disturbance signal into the node parameter field.

[0013] Preferably, the edge parameters in Sp3 include: The first parameter is the disturbance amplitude difference parameter A_diff, which is calculated from the absolute difference between the disturbance amplitudes of the two nodes. The second parameter, the rate of change difference parameter V_diff, is calculated from the absolute difference in the rates of change of the perturbations corresponding to the two nodes. The third parameter: the time interval parameter T_int, is calculated from the time difference between the occurrence of the two nodes; The three parameters are calculated according to the preset formula. The parameters for synthesizing edges are given, where k1, k2, and k3 are pre-defined weight constants.

[0014] Preferably, the topology constraint table includes: The node connection direction matrix M_dir is used to represent the allowed connection directions between node types; the edge parameter allowed range table M_rng is used to record the upper and lower limits of allowed edge parameters between different node type pairs; the node level table M_lvl is used to record the level number of the node type. Topology adjustments in SP5 are performed according to the constraints in the three tables mentioned above.

[0015] Preferably, the node merging rules in Sp4 include: Calculate the node dissimilarity value D_node between any two nodes; The D_node is obtained by summing the difference in disturbance amplitude, the difference in disturbance duration, and the difference in disturbance change rate according to a preset ratio; When D_node is less than the node merging threshold, the two nodes are merged into a new node, and the three types of perturbation parameters of the two nodes are averaged proportionally to obtain the parameters of the new node.

[0016] Preferably, the edge merging rules in Sp4 include: Sort the edges that connect pairs of the same nodes by their occurrence time; Select the first edge after sorting as the base edge; Calculate the difference in edge parameters for all candidate edges; When the difference value is less than the edge merging threshold, the candidate edge is merged into the base edge; The edge parameters of the base edge are updated to the weighted average of the parameters of all merged edges, with the weights calculated from the perturbation magnitudes of each edge.

[0017] Preferably, the cause-effect graph structure includes: The node list area N_liSpt stores the node type number and node parameters in order of node sequence. The edge list area E_liSpt records the starting node number, ending node number, and edge parameters of each edge in the form of triples. The topology description area T_deSpc records the topological relationships between nodes in a row-column format. The three components are combined in a fixed format to form the final output structure.

[0018] Preferably, all time parameters involved in Sp3 to Sp5 are generated based on a unified time base, which is constructed as follows: A unified clock is set up within the intelligent writing system; Each modal sampling module writes a timestamp of a unified clock at the sampling point; The time calculation for all node generation and edge establishment is based on this unified timestamp.

[0019] Preferably, when a certain disturbance signal is missing in Sp1, it is replaced according to the default parameter filling rule, which includes: The perturbation amplitude of the missing mode is assigned a preset default constant C_A; Assign a preset default rate constant C_V to the rate of change of the disturbance in the missing mode; The duration of the disturbance in the missing mode is assigned a preset time default constant C_T; The default constants are determined in advance by the device calibration dataset.

[0020] Preferably, the generation of the cause-effect graph structure maintains consistency regardless of the number of input modes, which is achieved through the following: Assign a modality identifier number to each modality; Set a modality identifier field in the node parameter structure to record the modality number from which the node originates; When generating the topology description area, the nodes are sorted according to the modality identifier field, so that the node arrangement does not depend on the number of input modalities.

[0021] Beneficial effects This invention provides a method for multimodal learning data acquisition and standardization based on an intelligent writing system. It has the following beneficial effects: 1. This invention introduces a unified time reference, perturbation parameter interval numbering, node type mapping table, modality identification field, and fixed-format node list area, edge list area, and topology description area into all acquisition modalities. This enables different sampling modalities to output standardized causal graphs in a unified structural form even when the number of inputs is inconsistent, the sampling frequency is different, or some modalities are missing. This allows the original multimodal signals to form a stable and consistent structured result after perturbation identification, node generation, edge construction, compression, and normalization. This avoids the problems of inconsistent structure, inconsistent dimensions, and changes in output format when fusing cross-modal data. It ensures that the data processing flow remains continuous and consistent under different learning tasks, different device configurations, and incomplete modal input conditions in practical application scenarios.

[0022] 2. This invention sets three distinct parameters—difference in perturbation amplitude, difference in rate of change, and node time interval—and constructs linear combination edge parameters using preset weights. Simultaneously, it utilizes a node difference formula to compress nodes, enabling quantifiable representations of multimodal learning behaviors at detailed levels such as extremely short-term perturbations, local velocity changes, minute pressure fluctuations, short-term gaze shifts, and speech amplitude changes. Furthermore, the structure is standardized through the connection direction matrix in the topological constraint table, the allowed range table of edge parameters, and the node hierarchy table. This allows the final causal graph to retain the temporal correlations and perturbation chain logic between learning behaviors with high parameter resolution, facilitating further processing in subsequent modules such as learning behavior modeling, writing process analysis, learning strategy recognition, or cognitive state inference. Attached Figure Description

[0023] Figure 1 A schematic diagram of the overall structure of the intelligent writing system of the present invention; Figure 2 A schematic diagram of the disturbance signal acquisition process of this invention; Figure 3 Flowchart of node generation in this invention; Figure 4 Flowchart of edge creation in this invention; Figure 5 A schematic diagram of the node merging rules of this invention; Figure 6 A schematic diagram of the edge merging rules of this invention; Figure 7 A schematic diagram of the topology normalization structure of this invention; Figure 8 A schematic diagram of the cause-effect graph structure output format of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Specific Implementation Example 1: like Figure 1-8As shown, this paper presents a method for multimodal learning data acquisition and standardization based on an intelligent writing system, and details its implementation. The system is deployed on an integrated hardware and software platform comprising a writing terminal and a server. The writing terminal consists of a touch-enabled tablet, a pressure-sensitive electromagnetic pen, a front-facing camera for capturing gaze coordinates, a microphone for capturing voice signals, and a clock chip providing a unified timestamp. The server is equipped with software modules for perturbation recognition, node generation, causal graph construction, and compression standardization. The terminal and server are connected via wired or wireless network. Raw sampled data can be preprocessed locally and then uploaded to the server for processes Sp1 to Sp6.

[0026] In this embodiment, the intelligent writing system collects raw data from multiple modalities during the learning process, including a handwriting coordinate sequence (x_i, y_i) collected at a sampling frequency of 200Hz, the corresponding pressure sampling sequence P_i, a 60Hz gaze coordinate sequence (g_x(j), g_y(j)) obtained through a gaze tracking algorithm, a short-time speech energy sequence E_k obtained by sampling at 16kHz and segmenting the frame, and a stroke continuity sequence obtained by whether the pen touches the screen. A unified clock module provides a millisecond-level timestamp t for all sampling points. Each modality appends this timestamp during sampling to construct a unified time reference. All subsequent time-related calculations (including disturbance duration, node time interval, edge time parameters, etc.) are based on this unified timestamp, which corresponds entirely to the construction of the "unified time reference." In the Sp1 "disturbance signal acquisition" step, the system first performs differential and thresholding on the above raw sequences to obtain disturbance signals belonging to different modalities. For the pressure sampling sequence P_i, the pressure change between adjacent sampling points is calculated: ; When |ΔP_i|>T_P, a pressure disturbance is considered to have occurred at time t_i, where T_P is a preset pressure change threshold. For the handwriting coordinate sequence, the instantaneous velocity is calculated. ; Then calculate the change in velocity: ; when At that time, the sampling point is marked as a velocity disturbance. For stroke continuity, the time interval between two strokes is calculated by detecting the process of pressure changing from non-zero to zero and then returning to zero. ,when At the corresponding moment, a "stroke interruption marker" perturbation is generated. The gaze coordinates are based on the gaze movement vectors of adjacent moments. ; And calculate the line-of-sight turning angle using the vector angle: ; ,when At that time, it was assumed that a "sudden angle change" perturbation occurred. The speech component first performed short-time energy calculations on the speech signal with a 20ms frame length and a 10ms frame shift, obtaining... Calculate the energy difference between frames: ; And in Speech amplitude perturbations are generated in real time. All these perturbations are encapsulated into a unified structure in the implementation, which includes at least: perturbation type (pressure, speed, interruption, gaze, speech), and perturbation occurrence timestamp. Disturbance amplitude (e.g.) The data includes the rate of change (amplitude change divided by the corresponding time interval) and the duration of the disturbance. This allows for the collection of all disturbance types described in Sp1 and the formation of a unified disturbance set.

[0027] In the Sp2 "node generation" step, the system implements the logic from Sp2-1 to Sp2-4. First, following Sp2-1, all disturbances obtained in the previous process are sorted by timestamp in ascending order, resulting in a time-ordered disturbance sequence. For any disturbance s in the sequence, the system divides the disturbance amplitude A_s into several amplitude intervals based on the predefined amplitude ranges of different modes. For example, the pressure disturbance amplitude interval is divided into [0, a_1), [a_1, a_2), ..., [a_{N_A-1}, a_{N_A}), and the amplitude interval number id_A is determined based on the interval A_s falls into. Similarly, the rate of change range is divided into several rate of change intervals, and the number id_V is obtained based on the interval where the disturbance rate of change V_s falls. The duration Dur_s is also divided into intervals to obtain the number id_T. This step corresponds to the calculation of "amplitude interval number, rate of change interval number, and duration interval number" in Sp2-2.

[0028] In its implementation, the node type mapping table is designed as a three-dimensional array NodeType[Na][Nv][Nt], which is logically equivalent to a mapping structure indexed by three numbers (id_A, id_V, id_T). The phrase "finding the corresponding row and column index in the node type mapping table based on the three numbers" described in Sp2-3 is specifically implemented in this embodiment as follows: using id_A as the first-dimensional index, id_V as the second-dimensional index, and id_T as the third-dimensional index, accessing NodeType[id_A][id_V][id_T] yields the node type number type_id. In Sp2-4, "determine the node type based on the index and write the three types of numbers of the disturbance signal into the node parameter field" is expressed in the program as follows: assign a new node ID to the disturbance, construct a node record Node, and fill in node_id, type_id, id_A, id_V, id_T and the disturbance timestamp t_node=t. At the same time, fill in the mode identifier field mode_id according to the disturbance source mode (for example, pressure disturbance is 1, speed disturbance is 2, line-of-sight disturbance is 3, voice disturbance is 4, and stroke interruption is 5), thus forming a complete node set, which is arranged in chronological order to prepare for Sp3.

[0029] During the Sp3 edge establishment process, the system calculates edge parameters. Let the time-ordered node set be {n_1, n_2, ..., n_N}, and the perturbation amplitude corresponding to node n_i be A_i, the rate of change be V_i, and the timestamp be t_i. The system sets an upper bound of a time window t_i + T_win for each node n_i, and scans all subsequent nodes n_j (t_j > t_i and t_j - t_i ≤ T_win) within this window. For each pair (i, j), three edge parameters are calculated: the perturbation amplitude difference parameter... ; Rate of change difference parameter: ; Time interval parameter: ; in and It can be the original disturbance amplitude, or it can be mapped to the interval median according to the interval number; and Similarly, the system establishes an edge for each pair of nodes that meet the conditions, recording the starting node index from_node_id=i, the ending node index to_node_id=j, and the comprehensive edge parameters obtained by combining the above three parameters through a preset calculation formula. A specific implementation of the synthesized edge parameters is a linear combination: ; in , , The constant weights set for the system before deployment based on the dimensions and importance of the business side, for example It mainly reflects the impact of amplitude difference. Reflecting the effect of the difference in rate of change, This reflects the effect of time intervals. Through this formula, the edge parameters... The three types of difference information are unified into a single real value. This parameter serves as both a feature value of the edge and a basis for filtering and truncation in subsequent edge merging, topological constraints, and graph structure output processes.

[0030] After constructing the initial edge set, proceed to Sp4 "structure compression". In the node merging section, for any two nodes... and The system is based on the defined node difference value. In this embodiment, the following formula can be used: ; in , , These represent the amplitude, rate of change, and duration of the disturbance corresponding to the node, respectively. , , This is the node difference weight constant. When the system determines that two nodes have a high degree of similarity, it merges them into a new node. The three types of parameters for the new node are calculated using a weighted average: ; in , It can be set to 1, or assigned different weights based on the node's source modality or frequency of occurrence. The node type number can be re-mapped based on the original types of the two nodes and the parameter range, or the type of one of the nodes can be retained. In the implementation, node merging is accomplished by traversing the node set and progressively updating the node list, while simultaneously adjusting the endpoints of the edges associated with the merged node. For any pair of nodes in the compressed node set... The system first collects the set of all edges connecting the node pair { Sort the edges by their starting time or the order in which they were created, and select the first edge. As a basic edge, the edge parameter of the basic edge is denoted as... For each candidate edge Calculate the difference in edge parameters: ; when When an edge is considered mergingable, it is added to the merge set. After completing the candidate traversal, the base edge parameters are updated to a weighted average of the parameters of all merged edges. ; in The set of indices of the edges to be merged. This can be set to the magnitude of the perturbation associated with the edge or other weight constants. After the update is complete, merged edges other than the base edges are removed from the edge set. After dual compression of nodes and edges, a new set of nodes and edges is obtained. In Sp5 "Topology Normalization", the system adjusts the compressed structure according to the topology constraint table. The topology constraint table includes the node connection direction matrix. Table of Allowable Ranges for Edge Parameters and node hierarchy table .in It is a Boolean value, when The time indicates the type is Nodes can be connected to types If the node types of the start and end nodes of an edge are respectively... and If the edge does not meet the direction constraint, it can be directly deleted in the implementation. For an interval Used to qualify type pairs The range of edge parameter values: when the edge parameter At that time, cut it into ;when At that time, cut it into . Record node type The hierarchical numbering, in scenarios where it is necessary to ensure unidirectional connections in the graph structure from lower to higher levels, can be combined with... Further filtering is performed to remove edges that do not conform to the hierarchical direction. This embodiment mainly uses... and The edge set is cleaned and pruned to ensure that the final causal graph structure meets the preset topological specifications.

[0031] Finally, in the Sp6 "Structure Output" step, the system organizes the normalized node set, edge set, and topological relationships into three structural areas: the node list area N_liSpt, the edge list area E_liSpt, and the topology description area T_deSpc. The node list area N_liSpt stores nodes in sequential order by node number. Each node record includes: node_id, node type number type_id, three interval numbers id_A, id_V, and id_T, a timestamp t_node, and a modality identifier field mode_id. To maintain consistency in the output structure when the number of input modalities varies, the system defines a fixed modality number for each modality and uses a sorting rule of "first by mode_id, then by t_node" when sorting nodes. Therefore, even if a modality is missing during a learning process, the remaining modality nodes are still arranged according to the consistent rule. The edge list area E_liSpt stores the starting node number, ending node number, and edge parameter P_{ij} of each edge in triplet form, corresponding one-to-one with the node numbers in the node list area. The topology description region T_deSpc records the connections between nodes using an adjacency matrix or adjacency list. In the adjacency matrix form, T_deSpc[i][j]=1 indicates that there is an edge between node i and node j, and the detailed parameters of the corresponding edge can be queried in E_liSpt. The entire causal graph structure only contains node number, node type, node parameter range number, modality number, edge connection relationship, and edge parameters, etc. It does not include direct numerical storage of the original pressure sequence, coordinate sequence, line-of-sight sequence, or speech energy sequence, and cannot recover the specific values ​​of the original perturbation signal sequence in Sp1 from it, thus satisfying the constraint of "structural output".

[0032] If a certain mode in Sp1 does not generate an identifiable disturbance or its corresponding data is missing, the system continues to execute each step according to the default parameter filling rules. Specifically, a constant C_{A} is assigned to the disturbance amplitude of the missing mode, a constant C_{V} is assigned to the disturbance change rate, and a constant C_{T} is assigned to the disturbance duration. These three constants can be set based on historical data statistics or experience before system deployment. When a missing mode exists, A_{i}, V_{i}, and Dur_{i} during node generation will use their corresponding default values ​​to participate in the calculations of Sp2 to Sp5, ensuring that the node structure fields are complete and the calculation formulas do not require branch judgments. The final output causal graph structure is consistent with the structure format when all modes are complete in terms of field dimensions.

[0033] Through the above embodiments, it can be clearly seen how each step derives a standardized causal graph structure consisting of three parts: N_liSpt, E_liSpt, and T_deSpc, from the original multimodal signal through perturbation detection, partition numbering, node type mapping, edge parameter calculation, structure compression, and topology normalization. Specific Implementation Example 2: Based on the technical solution of Specific Embodiment 1, the following technical content is further provided: In a real-world teaching scenario, a student completes the problem "Find a quadratic function" on a smart writing terminal. When the student reaches the "apex" of the writing and thinking process, the system performs unified acquisition of multimodal signals, disturbance identification, node generation, causal edge construction, structure compression, topology normalization, and causal graph structure output to achieve standardized modeling of learning behavior. The smart writing terminal includes a touch screen, an electromagnetic pen, an eye-tracking camera, a voice acquisition microphone, and a unified clock module. All sampled signals are appended with a unified millisecond timestamp during sampling. In this case, handwriting coordinates are sampled at 200Hz, pressure sequences are sampled synchronously with coordinates, eye-tracking coordinates are sampled at 60Hz, and voice signals are sampled at 16kHz and short-time energy is calculated in a 20ms frame length and 10ms frame shift mode. The entire multimodal raw data stream is input to the disturbance detection module on the server side to execute the Sp1 step. In Sp1, when the student writes the "recipe" derivation part (e.g., writing...), the system performs unified acquisition of multimodal signals, disturbance identification, node generation, causal edge construction, structure compression, topology normalization, and causal graph structure output to achieve standardized modeling of learning behavior. Convert to When the pressure sampling points changed from 180 to 245, the differential pressure sampling points showed a change. Exceeding the pressure threshold This is recorded as a pressure disturbance; when writing "(x-1)$2)^2" quickly, the handwriting speed increased from approximately 11.6 to 52.0, and the speed change Δv=40.4 exceeded the speed threshold T_V, which is recorded as a speed disturbance; when the student was thinking about "+1", there was a pause of 510ms, which exceeded the stroke interval threshold of 200ms, and was recorded as a stroke interruption disturbance; the gaze jumped between the question area and the draft area, and the angle between the gaze vectors reached 57°, which was greater than the 45° threshold, and was recorded as a gaze disturbance; speech energy frame difference A perturbation exceeding a speech threshold of 0.6 is recorded as a speech disturbance. A total of 27 disturbance points were generated during this writing process. The system then enters the Sp2 node generation stage. First, all disturbances are sorted by timestamp (Sp2-1). Then, three types of numbers are calculated for each disturbance (Sp2-2). For example, a pressure disturbance with an amplitude of 65 falls within the amplitude range [50, 100), with the number id_A=2; a rate of change of 1300 falls within the rate range [800, 2000), with the number id_V=2; and a duration of 5ms falls within the duration range [0, 10), with the number id_T=0. The system indexes the node type in the node type mapping table NodeType[id_A][id_V][id_T] using a triple (2, 2, 0), for example, mapping it to type_id=5 (Sp2-3), and generates the node structure Node. The time stamp, mode_id, is assigned a value based on the modality: 1=pressure, 2=speed, 3=interruption, 4=line of sight, 5=speech. All perturbations are mapped to form 27 time-ordered nodes. During the Sp3 setup step, the system sets a time window T_win=800ms for each node n_i, pairing all subsequent nodes n_j within the window that satisfy t_j-t_i≤800 to establish edges. Taking n5 and n9 as an example, their amplitude difference A_diff=|A9-A5|=30, their rate of change difference V_diff=400, and their time interval T_int=600ms. Based on the linear combination formula: ; If we set k1=1.0, k2=0.2, k3=0.001, then ; This results in the edge structure Edge{from=5, to=9, P=110.6}. A total of 89 edges are formed globally. During the Sp4 structure compression phase, the system uses the node difference formula: ; By performing pairwise comparisons of the nodes, for example, the amplitude difference between nodes 7 and 8 is 2, the duration difference is 3ms, and the rate difference is 20. Taking wA=1, wT=0.5, and wV=0.05, we obtain... ; Therefore, they are merged into a new node, and the parameters of the new node are as follows: Calculation, in this embodiment When compressing the edge set, if multiple edges connect the same pair of nodes, the base edges are selected in chronological order, and the differences in edge parameters are calculated. If it is less than the threshold Then add it to the merge set, and according to The basic edge parameters are updated, ultimately retaining only one edge. The compressed node set and edge set are then processed through Sp5 topology normalization. The system uses a node connection direction matrix. Check the start and end node types of each edge; if... If the edge is not found, delete it; then use the edge parameter allowable range table. Truncate parameters within a range; Node hierarchy table This provides hierarchical information to maintain structural consistency. After topology normalization, a final compliant causal graph structure is obtained. Finally, SP6 is executed, and the system outputs the node set to the node list area. (Including node_id, type_id, id_A, id_V, id_T, t_node, mode_id), output the edge set to an edge list. (from_node_id, to_node_id, P_ij) outputs the topological relationships as an adjacency matrix to This constitutes a complete standardized cause-effect graph data structure. Since all nodes have a `mode_id` field, and the output stage uses a fixed sorting format (first by `mode_id`, then by time order), in this case, if the gaze modality or speech modality is missing during a certain writing process, the system can still utilize default parameters. Fill in the corresponding fields without changing the output structure format, achieving structural consistency despite changes in the number of modalities. The final causal graph structure obtained in this case contains 21 compressed nodes, 47 normalized edges, and a 21×21 adjacency matrix TdeSpc. It fully reflects the temporal dependencies and perturbation causal chains of students' multimodal behaviors such as writing, thinking, hesitation, eye movement, and voice prompts during the "finding the vertex" derivation process. It can be directly applied to learning behavior analysis systems, cognitive state modeling modules, and teaching auxiliary decision-making modules.

[0035] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a reference structure" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0036] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for multimodal learning data acquisition and standardization based on an intelligent writing system, characterized by: Includes the following steps: Sp1: Disturbance signal acquisition: Acquire at least one disturbance signal generated by the intelligent writing system during the learning process. The disturbance signal includes pressure change in the pressure sampling sequence, speed change in the handwriting sampling sequence, interruption marker in the stroke continuity sequence, angle change in the line-of-sight coordinate sequence, or short-term amplitude change in the speech amplitude sequence. Sp2: Node generation: According to the preset disturbance classification rules, the collected disturbance signals are mapped to the node type mapping table according to their amplitude range, rate of change range and duration range, and a node set is generated based on the mapping results; Sp3: Edge Establishment: Within a preset time window, edges are established between nodes according to the order of their appearance and the numerical relationship of the corresponding disturbance signals, and edge parameters are calculated. The edge parameters are obtained by combining the disturbance amplitude difference, the rate of change difference, and the node time interval according to a preset calculation formula. Sp4: Structure Compression: Based on preset node merging and edge merging rules, the node set and edge set are compressed to generate a compressed node set and a compressed edge set. Sp5: Topology Normalization: Based on the preset topology constraint table, the compressed node set and edge set are topologically adjusted to conform to the connection direction, node hierarchy and edge parameter range requirements in the topology constraint table; Sp6: Structural Output: Outputs a causal graph structure containing a set of nodes, a set of edges, and topological description information, but does not contain numerical data that can be used to recover the original perturbation signal sequence in Sp1.

2. The method for multimodal learning data acquisition and standardization based on an intelligent writing system according to claim 1, characterized in that: Sp2 further includes: Sp2-1: Sort the disturbance signals according to the sampling time; Sp2-2: Calculate the interval number of disturbance amplitude, the interval number of change rate, and the interval number of duration for each disturbance signal; Sp2-3: Look up the corresponding row and column index in the node type mapping table based on the three numbers; Sp2-4: Determine the node type based on the index and write the three types of numbers of the disturbance signal into the node parameter field.

3. The method for multimodal learning data acquisition and standardization based on an intelligent writing system according to claim 1, characterized in that: The edge parameters in Sp3 include: The first parameter is the disturbance amplitude difference parameter A_diff, which is calculated from the absolute difference between the disturbance amplitudes of the two nodes. The second parameter, the rate of change difference parameter V_diff, is calculated from the absolute difference in the rates of change of the perturbations corresponding to the two nodes. The third parameter, the time interval parameter T_int, is calculated from the time difference between the occurrence of the two nodes.

4. The method for multimodal learning data acquisition and standardization based on an intelligent writing system according to claim 1, characterized in that: The topology constraint table includes: The node connection direction matrix M_dir is used to represent the allowed connection directions between node types; the edge parameter allowable range table M_rng is used to record the upper and lower limits of allowed edge parameters between different node type pairs; and the node level table M_lvl is used to record the level number of the node type.

5. The method for multimodal learning data acquisition and standardization based on an intelligent writing system according to claim 1, characterized in that: The node merging rules in Sp4 include: Calculate the node dissimilarity value D_node between any two nodes; The D_node is obtained by summing the difference in disturbance amplitude, the difference in disturbance duration, and the difference in disturbance change rate according to a preset ratio; When D_node is less than the node merging threshold, the two nodes are merged into a new node, and the three types of perturbation parameters of the two nodes are averaged proportionally to obtain the parameters of the new node.

6. The method for multimodal learning data acquisition and standardization based on an intelligent writing system according to claim 1, characterized in that: The edge merging rules in Sp4 include: Sort the edges that connect pairs of the same nodes by their occurrence time; Select the first edge after sorting as the base edge; Calculate the difference in edge parameters for all candidate edges; When the difference value is less than the edge merging threshold, the candidate edge is merged into the base edge; The edge parameters of the base edge are updated to the weighted average of the parameters of all merged edges, with the weights calculated from the perturbation magnitudes of each edge.

7. The method for multimodal learning data acquisition and standardization based on an intelligent writing system according to claim 1, characterized in that: The cause-effect graph structure includes: The node list area N_liSpt stores the node type number and node parameters in order of node sequence. The edge list area E_liSpt records the starting node number, ending node number, and edge parameters of each edge in the form of triples. The topology description area T_deSpc records the topological relationships between nodes in a row and column manner.

8. The method for multimodal learning data acquisition and standardization based on an intelligent writing system according to claim 1, characterized in that: All time parameters involved in Sp3 to Sp5 are generated based on a unified time base, which is constructed as follows: A unified clock is set up within the intelligent writing system; Each modal sampling module writes a timestamp of a unified clock at the sampling point; The time calculation for all node generation and edge establishment is based on this unified timestamp.

9. The method for multimodal learning data acquisition and standardization based on an intelligent writing system according to claim 1, characterized in that: When a certain disturbance signal is missing in Sp1, it is replaced according to the default parameter filling rules, which include: The perturbation amplitude of the missing mode is assigned a preset default constant C_A; Assign a preset default rate constant C_V to the rate of change of the disturbance in the missing mode; The duration of the disturbance in the missing mode is assigned a preset time default constant C_T.

10. The method for multimodal learning data acquisition and standardization based on an intelligent writing system according to claim 1, characterized in that: The generation of the cause-effect graph structure maintains consistency regardless of the number of input modes, which is achieved through the following: Assign a modality identifier number to each modality; Set a modality identifier field in the node parameter structure to record the modality number from which the node originates; When generating the topology description area, the nodes are sorted according to the modality identifier field, so that the node arrangement does not depend on the number of input modalities.

Images