Intelligent detection and management and control system for content of learning machine

Abstract

The invention relates to the technical field of computers, in particular to a learning practice machine content intelligent detection and management and control system, which comprises a touch track acquisition module used for acquiring pen wielding data in a writing process in real time; the multi-modal feature fusion module is used for judging whether the writing main body is in a learning state or not based on the dual-modal features; the dynamic attention analysis module is used for identifying an abnormal writing area based on a focusing modulation mechanism when the writing main body is in a non-learning state; the semantic understanding module is used for determining the semantic deviation degree of the semantic information and the learning text based on the semantic information recognition result of the written text in the abnormal writing area; and the hierarchical intervention module is used for triggering corresponding intervention measures in a hierarchical manner based on the semantic deviation degree. The detection precision of the abnormal behavior of the writing main body is improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to an intelligent detection and control system for learning and training machine content. Background Technology

[0002] With the widespread adoption of smart educational devices, learning and practice machines have become essential tools for students' daily learning. However, existing devices lack the ability to intelligently monitor draft content. Traditional methods mainly rely on post-processing manual checks or simple trajectory analysis, making it difficult to identify abnormal behaviors such as scribbling and distracted writing in real time. Particularly in multimodal feature fusion and dynamic attention analysis, existing technologies suffer from insufficient feature extraction and inadequate contextual understanding, limiting detection accuracy and real-time performance. Most current systems employ fixed threshold intervention strategies, lacking intelligent hierarchical control mechanisms based on semantic understanding.

[0003] Chinese Patent Application Publication No. CN110634337A discloses a method and system for controlling an intelligent early education machine. This system can import a database of learning needs based on user requirements, increasing the selectivity of learning content. The intelligent early education machine can arrange learning courses according to the user's spare time, allowing for systematic learning. It records video and audio information during the user's learning process, helping parents correct and guide their children's learning while also recording the child's learning video data. The system allows for interaction with the child through the intelligent early education machine or external intelligent devices connected to it, increasing the child's motivation to learn.

[0004] The following problems still exist in the existing technology: (1) Traditional trajectory analysis cannot integrate spatiotemporal multidimensional features, resulting in a high rate of false negatives in abnormal behavior detection; (2) Existing attention mechanisms are not sensitive enough to nonlinear trajectories and edge regions, making it difficult to adapt to dynamic writing scenarios; (3) Intervention strategies are disconnected from semantic understanding, making it impossible to implement differentiated management based on content relevance. Summary of the Invention

[0005] To address these issues, this invention provides an intelligent detection and control system for learning and practice machine content, which overcomes the problems of high false negative rate in abnormal behavior detection, difficulty in adapting to dynamic writing scenarios, and inability to implement differentiated control based on content relevance in existing technologies.

[0006] To achieve the above objectives, the present invention provides an intelligent detection and control system for learning and practice machine content, comprising:

[0007] The touch trajectory acquisition module is used to collect pen movement data in real time during the writing process;

[0008] A multimodal feature fusion module, which is connected to the touch trajectory acquisition module, is used to determine whether the writing subject is in a learning state based on the pen trajectory image generated by fusing the pen movement data and the dual-modal features generated by the pen movement dynamic features.

[0009] A dynamic attention analysis module, which is connected to the multimodal feature fusion module, is used to identify abnormal writing areas based on a focus modulation mechanism when the writing subject is in a non-learning state.

[0010] A semantic understanding module, which is connected to the dynamic attention analysis module, is used to determine the semantic deviation between the semantic information and the learned text based on the semantic information recognition results of the written text in the writing abnormal region.

[0011] The graded intervention module is connected to the semantic understanding module and the dynamic attention analysis module respectively, and is used to trigger corresponding intervention measures based on the semantic deviation grade.

[0012] Furthermore, the touch trajectory acquisition module uses an electromagnetic induction layer to capture the handwriting coordinate sequence;

[0013] Pressure data is recorded using a pressure sensor;

[0014] Based on the handwriting coordinate sequence and pressure-sensitive data, an N-row, 5-column spatiotemporal feature matrix of pen movement data is established. The spatiotemporal feature matrix includes five-dimensional features: x and y coordinates, pen pressure, timestamp, and pen angle. N is the number of handwriting points.

[0015] Furthermore, the multimodal feature fusion module includes;

[0016] The spatial feature extraction unit is used to extract P3-P5 multi-scale features of the pen trajectory image using a depthwise separable convolution model, wherein the pen trajectory image is the line connecting the positions of continuous coordinate points in the pen data.

[0017] A temporal feature extraction unit is used to analyze the pen movement data through a BiLSTM network to generate dynamic pen movement features, wherein the dynamic pen movement features include speed fluctuation amplitude and pressure fluctuation frequency.

[0018] The feature fusion unit is used to dynamically and weightedly fuse the multi-scale features and the pen movement dynamic features using a gated attention mechanism to form bimodal features.

[0019] Furthermore, the multimodal feature fusion module outputs a determination result on whether the writing subject is in a learning state based on the dual-modal features.

[0020] Furthermore, the dynamic attention analysis module is used to construct a dynamic attention region based on the learned text;

[0021] Spatial pyramid pooling is used to extract multi-scale contextual features of the dynamically interested region;

[0022] The multi-scale context features are modulated using a focused modulation mechanism, and the modulation process includes hierarchical feature gating aggregation and query adaptive modulation.

[0023] Identify abnormal writing regions based on modulation results;

[0024] The hierarchical feature gating aggregation formula is Zout = Σ(Gl ⊙ Zl), where Zout is the hierarchical feature gating aggregation result, Zl is the feature map of different levels, Gl is the corresponding gating weight, and ⊙ is the element-wise multiplication operation.

[0025] The query adaptive modulation formula is yi = q(xi) ⊙ h(Zout), where yi is the query adaptive modulation result, xi is the original feature vector of the i-th spatial position in the input feature map, q(xi) is the query function, and h(Zout) is the modulation function.

[0026] Furthermore, based on the identified writing anomaly regions, the semantic understanding module uses a lightweight Transformer architecture to identify the semantic deviation between the written text in the writing anomaly regions and the learned text.

[0027] Furthermore, the graded intervention module determines to trigger a level one response and provides visual cues based on the comparison result that the semantic deviation is less than a first preset deviation.

[0028] The graded intervention module determines to trigger a secondary response based on the comparison result that the semantic deviation is greater than or equal to the first preset deviation and less than the second preset deviation, and automatically fades irrelevant handwriting.

[0029] The graded intervention module determines to trigger a level 3 response based on the comparison result that the semantic deviation is greater than the second preset deviation, locks the writing area, and generates a behavior report to be sent to the supervisory end.

[0030] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention constructs a five-dimensional spatiotemporal feature matrix through a 200Hz high-precision trajectory acquisition module, and combines multi-scale image features of the multi-modal feature fusion module with BiLSTM temporal feature analysis to comprehensively characterize writing behavior; in conjunction with the improved focusing modulation mechanism of the dynamic attention analysis module, it enhances the expression of abnormal features in edge regions and nonlinear trajectories, effectively reduces the false negative rate of abnormal behavior detection, and improves detection accuracy.

[0031] Furthermore, this invention constructs a dynamic attention region template based on course content and combines it with the multi-scale context awareness capability of spatial pyramid pooling, enabling the system to adapt to different dynamic writing scenarios. This improves the sensitivity to detecting nonlinear trajectories and abnormal behaviors in edge regions, solving the problem that existing technologies struggle to adapt to dynamic writing scenarios.

[0032] Furthermore, this invention implements semantic correlation analysis through a lightweight Transformer architecture, establishes a graded intervention strategy based on deviation, and implements differentiated control measures according to the degree of anomaly. This avoids the blindness of fixed threshold intervention, reduces invalid alarms, improves the intervention effect, and at the same time protects the students' learning experience. Attached Figure Description

[0033] Figure 1 This is a structural block diagram of the intelligent detection and control system for learning and practice machine content according to an embodiment of the present invention;

[0034] Figure 2 This is a flowchart illustrating the triggering of tiered intervention measures in an embodiment of the present invention. Detailed Implementation

[0035] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0036] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0037] Please see Figures 1-2 As shown, Figure 1 This is a structural block diagram of the intelligent detection and control system for learning and practice machine content according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the triggering of tiered intervention measures in an embodiment of the present invention.

[0038] The present invention provides an intelligent detection and control system for learning and practice machine content, comprising:

[0039] The touch trajectory acquisition module is used to collect pen movement data in real time during the writing process;

[0040] A multimodal feature fusion module, which is connected to the touch trajectory acquisition module, is used to determine whether the writing subject is in a learning state based on the pen trajectory image generated by fusing the pen movement data and the dual-modal features generated by the pen movement dynamic features.

[0041] A dynamic attention analysis module, which is connected to the multimodal feature fusion module, is used to identify abnormal writing areas based on a focus modulation mechanism when the writing subject is in a non-learning state.

[0042] A semantic understanding module, which is connected to the dynamic attention analysis module, is used to determine the semantic deviation between the semantic information and the learned text based on the semantic information recognition results of the written text in the writing abnormal region.

[0043] The graded intervention module is connected to the semantic understanding module and the dynamic attention analysis module respectively, and is used to trigger corresponding intervention measures based on the semantic deviation grade.

[0044] Specifically, the touch trajectory acquisition module uses an electromagnetic induction layer to capture handwriting coordinate sequences;

[0045] Pressure data is recorded using a pressure sensor;

[0046] Based on the handwriting coordinate sequence and pressure-sensitive data, an N-row, 5-column spatiotemporal feature matrix of pen movement data is established. The spatiotemporal feature matrix includes five-dimensional features: x and y coordinates, pen pressure, timestamp, and pen angle. N is the number of handwriting points.

[0047] Specifically, pen movement data includes pen position, pen pressure, timestamp, and pen angle to ensure a high-quality data foundation for subsequent analysis.

[0048] The collected pen stroke data is stored in matrix form X∈R(N×5), where X represents the matrix storing the writing data, R(N×5) indicates that the matrix has N rows and 5 columns, and N represents the number of pen strokes recorded. For example, if 100 strokes are written, there will be 100 rows, and the 5 columns correspond to:

[0049] x\y coordinates: the position of the pen tip on the screen, such as (120, 300) which means 120 pixels horizontally and 300 pixels vertically;

[0050] Pen pressure: The pressure applied when pressing the pen tip.

[0051] Timestamp: The time of each entry, such as 16:30:25.123.

[0052] Pen angle: the direction in which the pen is tilted, such as 45 degrees, which means tilting to the upper right.

[0053] For example, when the writing subject writes the character 'one', the spatio-temporal feature matrix recorded by the touch trajectory acquisition module is: [(50, 200, 1024, 16:30:00, 30), (150, 200, 2048, 16:30:01, 25)]. This matrix indicates that the pen tip moves from the position (50, 200) to the position (150, 200), the pressure value gradually increases from 1024 to 2048, it takes 1 second, and the writing angle changes from 30 degrees to 25 degrees.

[0054] Specifically, the multi-modal feature fusion module includes;

[0055] A spatial feature extraction unit, which is used to extract the P3-P5 multi-scale features of the writing trajectory image using a depthwise separable convolution model. Among them, the writing trajectory image is the position connection of consecutive coordinate points in the writing data;

[0056] A temporal feature extraction unit, which is used to analyze the writing data through a BiLSTM network to generate writing dynamic features. Among them, the writing dynamic features include the speed fluctuation amplitude and the pressure fluctuation frequency;

[0057] A feature fusion unit, which is used to dynamically weight and fuse the multi-scale features and the writing dynamic features using a gated attention mechanism to form bimodal features.

[0058] Specifically, P3, P4, and P5 correspond to the feature maps of 8-fold, 16-fold, and 32-fold downsampling respectively. The P3 feature map has a higher resolution and can be used to capture details such as line contours; the P5 feature map has a lower resolution and can be used to identify the global information of the overall pattern; the P4 feature map is in between, realizing the transition between details and global information.

[0059] Example: When the writing subject draws a triangle on the screen:

[0060] P3 layer: Identify the stroke quality of the three sides and whether they are continuous;

[0061] P4 layer: Confirm the basic shape of the triangle;

[0062] P5 layer: Judge the positional relationship of the triangle in the draft area.

[0063] Specifically, the BiLSTM network can simultaneously utilize forward and backward temporal information to accurately capture patterns such as speed changes and pauses during pen strokes. For example, the pen stroke speed is stable when solving problems, but fluctuates significantly when doodling. By calculating the instantaneous speed of the pen stroke through continuous timestamps, the pen stroke speed is stable at around 15mm / s when the writing subject is solving problems normally, with small fluctuations in pen stroke speed and smooth changes in pen stroke acceleration. The pen stroke pressure value is stable above 2000 levels when solving problems, while the speed fluctuates drastically when the writing subject is distracted and doodling, ranging from 5mm / s to 25mm / s, with frequent abrupt changes in pen stroke acceleration and irregular fluctuations in pen stroke pressure value between 800 and 1200 levels.

[0064] Specifically, the feature fusion unit uses a gated attention mechanism to dynamically weight and fuse features to generate bimodal features. This unit learns the importance weights of the two features in different scenarios to achieve adaptive feature fusion. For example, when determining whether it is problem-solving writing, trajectory image features, such as formula outlines, are assigned higher weights, while pen movement temporal features, such as stable speed, are assigned auxiliary weights. When determining whether it is scribbling, pen movement temporal features, such as pen movement acceleration, are assigned higher weights, ultimately forming a bimodal feature matrix.

[0065] Specifically, the multimodal feature fusion module outputs a determination result on whether the writing subject is in a learning state based on the dual-modal features.

[0066] Specifically, the dynamic attention analysis module is used to construct dynamic attention regions based on the learned text;

[0067] Spatial pyramid pooling is used to extract multi-scale contextual features of the dynamically interested region;

[0068] The multi-scale context features are modulated using a focused modulation mechanism, and the modulation process includes hierarchical feature gating aggregation and query adaptive modulation.

[0069] Identify abnormal writing regions based on modulation results;

[0070] Among them, the hierarchical feature gating aggregation formula is Zout = Σ(Gl ⊙ Zl);

[0071] The adaptive modulation formula is: yi = q(xi) ⊙ h(Zout);

[0072] Wherein, Zout is the hierarchical feature gating aggregation result, Gl is the gating weight, used to control the contribution of different level feature maps (Zl), such as assigning a higher Gl value to the feature map corresponding to the graffiti area and a lower Gl value to the normal writing area; Zl is the P3-P5 multi-scale feature map obtained by the spatial feature extraction branch; ⊙ is the element-wise multiplication operation to achieve feature filtering and enhancement. yi is the query adaptive modulation result, xi is the original feature vector of the i-th spatial position in the input feature map; q(xi) is the query function, used to locate the current handwriting position, such as the coordinate position of detected graffiti behaviors such as "drawing a little person" or "random lines"; h(Zout) is the modulation function, used to adjust the intensity of the aggregated feature (Zout), amplifying the abnormal features of the graffiti area and suppressing the features of the normal writing area.

[0073] To enhance the monitoring effect of edge regions, this module sets the gate weight of edge regions to G1=0.3 and the gate weight of core regions to G2=0.7. At this time, the calculation formula of hierarchical feature gating aggregation can be specified as Zout = 0.3×Zedge +0.7×Zcenter, where Zedge is the feature map of the edge region and Zcenter is the feature map of the core region. This setting improves the detection sensitivity of abnormal behavior in edge regions.

[0074] Specifically, the dynamic attention analysis module predicts reasonable writing areas based on the current learning text. For example, during math practice, it sets the areas in the draft area where formulas and problem-solving steps may be written as core attention areas, and other areas as peripheral attention areas.

[0075] Specifically, the semantic understanding module, based on a determined writing anomaly region, uses a lightweight Transformer architecture to identify the semantic deviation between the written text in the writing anomaly region and the learned text.

[0076] Specifically, the semantic understanding module uses a lightweight Transformer architecture to identify the semantic correlation between doodle patterns and text content, determining whether the written content is relevant to the current learning task. The lightweight Transformer architecture has the advantages of fewer parameters and faster processing speed, making it suitable for deployment on embedded devices such as learning machines. Its implementation process is as follows: abnormal writing region features output by the dynamic attention analysis module are input into the lightweight Transformer; the encoder extracts semantic features, which are then matched against a pre-defined semantic database of course content to calculate the semantic deviation.

[0077] Specifically, the graded intervention module determines to trigger a level one response and provides visual cues based on the comparison result that the semantic deviation is less than a first preset deviation.

[0078] The graded intervention module determines to trigger a secondary response based on the comparison result that the semantic deviation is greater than or equal to the first preset deviation and less than the second preset deviation, and automatically fades irrelevant handwriting.

[0079] The graded intervention module determines to trigger a level 3 response based on the comparison result that the semantic deviation is greater than the second preset deviation, locks the writing area, and generates a behavior report to be sent to the supervisory end.

[0080] Specifically, the first preset deviation is 30%, and the second preset deviation is 60%.

[0081] Specifically, when the semantic deviation is less than 30%, it is judged as slight distraction, triggering a visual cue icon, such as a flashing "Focus on Learning" icon, to gently remind students to focus and avoid excessive intervention that could disrupt their learning rhythm. When the semantic deviation is between 30% and 60%, it is judged as moderate distraction, automatically fading irrelevant handwriting so that students cannot clearly see the doodles, while retaining writing content related to learning to guide students back to their learning tasks. When the semantic deviation is ≥ 60%, it is judged as severe distraction, locking the writing area and prohibiting students from continuing to write in the area. At the same time, a behavior report is generated, including information such as the time of distraction and the type of distracted content, and sent to the teacher or parent so that teachers and parents can intervene and guide them in a timely manner.

[0082] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A learning and practice machine content intelligent detection and control system, characterized in that, include: The touch trajectory acquisition module is used to collect pen movement data in real time during the writing process; A multimodal feature fusion module, which is connected to the touch trajectory acquisition module, is used to determine whether the writing subject is in a learning state based on the pen trajectory image generated by fusing the pen movement data and the dual-modal features generated by the pen movement dynamic features. A dynamic attention analysis module, which is connected to the multimodal feature fusion module, is used to identify abnormal writing areas based on a focus modulation mechanism when the writing subject is in a non-learning state. A semantic understanding module, which is connected to the dynamic attention analysis module, is used to determine the semantic deviation between the semantic information and the learned text based on the semantic information recognition results of the written text in the writing abnormal region. The graded intervention module is connected to the semantic understanding module and the dynamic attention analysis module respectively, and is used to trigger corresponding intervention measures based on the semantic deviation grade.

2. The intelligent detection and control system for learning and practice machine content according to claim 1, characterized in that, The touch trajectory acquisition module uses an electromagnetic induction layer to capture handwriting coordinate sequences; Pressure data is recorded using a pressure sensor; Based on the handwriting coordinate sequence and pressure-sensitive data, an N-row, 5-column spatiotemporal feature matrix of pen movement data is established. The spatiotemporal feature matrix includes five-dimensional features: x and y coordinates, pen pressure, timestamp, and pen angle. N is the number of handwriting points.

3. The intelligent detection and control system for learning and practice machine content according to claim 1, characterized in that, The multimodal feature fusion module includes: The spatial feature extraction unit is used to extract P3-P5 multi-scale features of the pen trajectory image using a depthwise separable convolution model, wherein the pen trajectory image is the line connecting the positions of continuous coordinate points in the pen data. A temporal feature extraction unit is used to analyze the pen movement data through a BiLSTM network to generate dynamic pen movement features, wherein the dynamic pen movement features include speed fluctuation amplitude and pressure fluctuation frequency. The feature fusion unit is used to dynamically and weightedly fuse the multi-scale features and the pen movement dynamic features using a gated attention mechanism to form bimodal features.

4. The intelligent detection and control system for learning and practice machine content according to claim 3, characterized in that, The multimodal feature fusion module outputs a determination result on whether the writing subject is in a learning state based on the bimodal features.

5. The intelligent detection and control system for learning and practice machine content according to claim 4, characterized in that, The dynamic attention analysis module is used to construct dynamic attention regions based on the learned text; Spatial pyramid pooling is used to extract multi-scale contextual features of the dynamically interested region; The multi-scale context features are modulated using a focused modulation mechanism, and the modulation process includes hierarchical feature gating aggregation and query adaptive modulation. Identify abnormal writing regions based on modulation results; The hierarchical feature gating aggregation formula is Zout = Σ(Gl ⊙ Zl), where Zout is the hierarchical feature gating aggregation result, Zl is the feature map of different levels, Gl is the corresponding gating weight, and ⊙ is the element-wise multiplication operation. The formula for querying adaptive modulation is yi = q(xi) ⊙ h(Zout), where yi is the query adaptive modulation result, xi is the original feature vector of the i-th spatial position in the input feature map, q(xi) is the query function, and h(Zout) is the modulation function.

6. The intelligent detection and control system for learning and practice machine content according to claim 5, characterized in that, The semantic understanding module, based on the identified writing anomaly regions, uses a lightweight Transformer architecture to identify the semantic deviation between the written text in the writing anomaly regions and the learned text.

7. The intelligent detection and control system for learning and practice machine content according to claim 6, characterized in that, The graded intervention module determines to trigger a level one response and provides visual cues based on the comparison result that the semantic deviation is less than a first preset deviation. The graded intervention module determines to trigger a secondary response based on the comparison result that the semantic deviation is greater than or equal to the first preset deviation and less than the second preset deviation, and automatically fades irrelevant handwriting. The graded intervention module determines to trigger a level 3 response based on the comparison result that the semantic deviation is greater than the second preset deviation, locks the writing area, and generates a behavior report to be sent to the supervisory end.

Images