Robot teaching system for children based on somatosensory interaction and device

Abstract

The application relates to the technical field of robot teaching, and discloses a robot infant teaching system and equipment based on somatosensory interaction. A somatosensory data acquisition module acquires skeletal point motion time sequence data and interactive behavior video streams of a target infant through a depth camera and an inertial sensor; a multi-modal feature fusion module extracts dynamic trajectory features from the skeletal motion data, analyzes behavior semantic features from the video streams, and generates a fused interactive feature set through cross-modal correlation fusion; an interaction sensitivity calculation module calls a pre-trained dynamic weighting network model to perform attention allocation on the fused features, and generates a teaching interaction sensitivity coefficient set containing response sensitivity distributions of different teaching links; an abnormal behavior tracing module generates a teaching behavior abnormality tracing result indicating an abnormal source link and a deviation type; and an interaction strategy generation module generates an adaptive teaching strategy according to the tracing result and sends the adaptive teaching strategy to a robot motion control unit.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot teaching, in particular to a robot infant teaching system and device based on somatosensory interaction. BACKGROUND

[0002] In the field of infant education, with the development of intelligent technology, robot teaching systems have gradually become an important tool to assist infants in learning. Most existing infant teaching robots rely on pre-set programs to carry out teaching activities, lacking dynamic perception and flexible response to the real-time behavior of infants. Although some systems attempt to introduce interaction functions, there are obvious limitations in behavior perception and teaching adaptation.

[0003] Most robot teaching systems have single sensing means, often only collecting image information of infants through ordinary cameras or only obtaining interaction signals through simple touch sensors. This single-modal sensing method cannot fully capture the motion state and behavior intention of infants, such as accurately identifying the amplitude of infant's body movements, the continuity of movements, and the details of expression changes. Due to the lack of precise tracking of skeletal movement trajectories, the system cannot determine whether the infant actively participates in the teaching activity, nor can it perceive the motion standardization of the infant during the operation process.

[0004] In terms of feature processing, existing technologies lack effective fusion analysis of collected raw data. Image information and simple motion signals are often processed separately, without establishing a correlation between different modal data, resulting in feature vectors that cannot fully reflect the interaction state of infants. This causes the teaching system to have deviations in understanding the behavior semantics of infants, making it difficult to distinguish between unintentional actions and intentional interactions of infants, thereby affecting the accuracy of teaching responses.

[0005] Existing systems lack dynamic adaptability in interaction sensitivity adjustment. During the teaching process, different stages have different needs for interaction response, such as higher sensitivity in the game stage to enhance interest, and appropriate reduction of sensitivity in the knowledge explanation stage to avoid interference. However, existing systems mostly use fixed response thresholds, which cannot adjust the sensitivity distribution according to the real-time behavior characteristics of infants, easily leading to problems of over-response or lagging response.

[0006] When infants exhibit abnormal behaviors such as distraction of attention and deviation of actions from teaching requirements, existing systems have difficulty in quickly tracing the source of abnormalities. Due to the lack of joint analysis of skeletal movement timing data and behavior video streams, it is difficult to locate the specific teaching stage where the abnormal behavior occurs, and it is also difficult to judge the type of behavior deviation, resulting in a lack of targetedness in teaching intervention. In terms of strategy generation, the teaching strategies of existing systems are mostly pre-set, and cannot be dynamically adjusted according to the real-time behavior abnormalities of infants, making it difficult to meet the individualized learning needs of different infants and affecting the improvement of teaching effectiveness. SUMMARY

[0007] The present application aims to provide a somatosensory interaction-based robot infant teaching system to solve the problems raised in the background art.

[0008] To achieve the above-mentioned purpose, the present application provides a somatosensory interaction-based robot infant teaching system, which comprises:

[0009] A somatosensory data acquisition module is configured to acquire skeletal point motion time series data and interactive behavior video stream of a target infant through a depth camera and an inertial sensor;

[0010] A multi-modal feature fusion module is configured to perform dynamic trajectory feature extraction processing on the skeletal point motion time series data to generate a skeletal motion feature vector, perform action semantic analysis processing on the interactive behavior video stream to generate a behavior semantic feature vector, and perform cross-modal correlation fusion on the skeletal motion feature vector and the behavior semantic feature vector to generate a fused interactive feature set;

[0011] An interaction sensitivity calculation module is configured to call a pre-trained dynamic weighting network model to perform attention allocation processing on the fused interactive feature set to generate a teaching interaction sensitivity coefficient set, which contains interactive response sensitivity distribution corresponding to different teaching links;

[0012] An abnormal behavior tracing module is configured to perform joint root cause analysis on the skeletal point motion time series data and the interactive behavior video stream based on the interactive response sensitivity distribution to generate a teaching behavior abnormality tracing result, which is used to indicate the teaching link and behavior deviation type of the abnormal source;

[0013] An interaction strategy generation module is configured to generate an adaptive teaching strategy according to the teaching behavior abnormality tracing result and send the adaptive teaching strategy to a robot motion control unit.

[0014] Preferably, the action semantic analysis processing on the interactive behavior video stream to generate a behavior semantic feature vector comprises:

[0015] Performing action sequence segmentation processing on the interactive behavior video stream to obtain a plurality of continuous action segments;

[0016] Calling a pre-trained three-dimensional convolutional network model to perform spatio-temporal feature encoding processing on each of the continuous action segments to generate an initial action feature vector;

[0017] The initial action feature vector is subjected to teaching behavior feature enhancement processing, including: matching a standard behavior template set associated with the current action segment from a preset infant behavior knowledge base; inputting the standard behavior template set into a feature encoder for template semantic embedding processing to generate a template feature vector set; and performing similarity weighted fusion on the template feature vector set and the initial action feature vector to generate an enhanced action feature vector.

[0018] The enhanced action feature vectors of each continuous action segment are subjected to time sequence position coding integration processing to generate the behavior semantic feature vector.

[0019] Preferably, the dynamic trajectory feature extraction processing on the skeletal point motion time sequence data generates a skeletal motion feature vector, including:

[0020] The skeletal point motion time sequence data is subjected to abnormal action interval detection processing to identify an abnormal fluctuation time segment in the joint angle change sequence;

[0021] The original skeletal data is subjected to multi-scale sliding window sampling processing within the abnormal fluctuation time segment to obtain a plurality of local trajectory data segments;

[0022] A pre-trained graph convolution network model is called to perform joint correlation feature extraction processing on each of the local trajectory data segments to generate a local motion feature vector;

[0023] The local motion feature vectors of each of the local trajectory data segments are subjected to global trajectory aggregation processing to generate the skeletal motion feature vector.

[0024] Preferably, the cross-modal correlation fusion of the skeletal motion feature vector and the behavior semantic feature vector generates a fusion interaction feature set, including:

[0025] The behavior semantic feature vector is subjected to time dimension alignment processing to map it to the same timestamp sequence as the skeletal motion feature vector;

[0026] A skeletal behavior cross-attention mechanism is constructed to calculate a cross-modal correlation matrix between the trajectory feature of each timestamp in the skeletal motion feature vector and the semantic feature of the corresponding timestamp in the behavior semantic feature vector;

[0027] Based on the cross-modal correlation matrix, bidirectional feature interaction processing is performed on the skeletal motion feature vector and the behavior semantic feature vector to generate an interaction trajectory feature vector and an interaction semantic feature vector;

[0028] The interaction trajectory feature vector and the interaction semantic feature vector are subjected to gated feature fusion processing to generate the fusion interaction feature set.

[0029] Preferably, the pre-trained dynamic weighting network model is called to perform attention allocation processing on the fusion interaction feature set to generate a teaching interaction sensitivity coefficient set, including:

[0030] The fusion interaction feature set is input into the feature screening layer of the dynamic weighting network model for key feature extraction processing to obtain a screened core interaction feature set;

[0031] The cross-correlation layer of the dynamic weighting network model is called to perform high-order feature combination processing on the core interaction feature set to generate an associated feature tensor;

[0032] The sensitivity distribution layer of the dynamic weighting network model is called to perform teaching session weight calculation processing on the associated feature tensor to generate an interaction response sensitivity distribution of different teaching sessions;

[0033] Based on the interaction response sensitivity distribution, the associated feature tensor is subjected to feature weighting and aggregation processing to generate the teaching interaction sensitivity coefficient set.

[0034] Preferably, the joint root cause analysis is performed on the skeletal point motion time series data and the interaction behavior video stream based on the interaction response sensitivity distribution to generate a teaching behavior abnormality traceability result, including:

[0035] According to the interaction response sensitivity distribution, an abnormal sensitivity item exceeding a preset threshold is screened to generate a candidate abnormal feature set;

[0036] Each abnormal sensitivity item in the candidate abnormal feature set is subjected to reverse feature analysis processing to determine its corresponding skeletal motion abnormality description and behavior semantic deviation description;

[0037] A pre-trained traceability inference model is called to perform joint causal analysis processing on the skeletal motion abnormality description and the behavior semantic deviation description to generate an abnormal traceability path set;

[0038] The abnormal traceability path set is subjected to confidence evaluation processing to screen a target traceability path with the highest confidence as the teaching behavior abnormality traceability result.

[0039] Preferably, the adaptive teaching strategy is generated according to the teaching behavior abnormality traceability result, including:

[0040] The abnormal source teaching session and the behavior deviation type in the teaching behavior abnormality traceability result are analyzed;

[0041] A historical adjustment strategy set associated with the abnormal source teaching session is matched from a historical teaching strategy library;

[0042] Screening the historical adjustment strategy set based on the behavior deviation type to obtain an effective interaction strategy set;

[0043] Optimizing the effective interaction strategy set to generate an adaptive teaching strategy meeting teaching constraint conditions.

[0044] Preferably, the optimizing the effective interaction strategy set to generate an adaptive teaching strategy meeting teaching constraint conditions comprises:

[0045] Constructing a three-dimensional optimization target space containing teaching progress completion rate, behavior deviation correction rate and interaction response timeliness;

[0046] Mapping each effective interaction strategy to a coordinate point in the three-dimensional optimization target space;

[0047] Screening a candidate strategy set located on a Pareto front through a multi-objective optimization algorithm;

[0048] Calling a strategy adaptation model to perform adaptive scoring processing on the candidate strategy set based on the real-time state of a current teaching scene;

[0049] Selecting a candidate strategy with the highest adaptive score as the adaptive teaching strategy.

[0050] Preferably, the system further comprises:

[0051] A real-time feedback module configured to continuously collect physiological response data and behavior feedback data of the infant during execution of the adaptive teaching strategy by the robot;

[0052] A strategy optimization module configured to input the physiological response data and behavior feedback data into an incremental learning model to perform dynamic strategy optimization processing and generate an optimized adaptive teaching strategy.

[0053] Preferably, the strategy optimization module performs dynamic strategy optimization processing, comprising:

[0054] Performing correlation analysis processing on execution effect data of the current adaptive teaching strategy and the physiological response data to generate a strategy correction coefficient;

[0055] Performing dynamic adjustment processing on interaction response parameters in the adaptive teaching strategy based on the strategy correction coefficient;

[0056] Performing matching verification processing on the adjusted interaction response parameters and the behavior feedback data to generate the optimized adaptive teaching strategy.

[0057] Compared with the prior art, the present application has the following advantages:

[0058] The somatosensory interaction-based robot early childhood education system effectively makes up for the shortcomings of existing early childhood education robots in behavior perception, feature processing, response adjustment, and strategy adaptation through the cooperative work of multiple modules.

[0059] At the data acquisition level, the system uses a combination of depth cameras and inertial sensors to synchronously acquire skeletal point motion timing data and interactive behavior video streams of children, breaking through the perception limitations of single sensors. This multi-source data acquisition mode can comprehensively capture children's limb movement trajectories, action rhythms, and behavior scene information, providing rich raw data support for subsequent feature analysis, making the system's perception of children's behavior more comprehensive and detailed.

[0060] The multi-modal feature fusion module processes skeletal motion data and behavior video streams in a targeted manner and generates a set of fused interactive features through cross-modal association fusion. This process overcomes the shortcomings of isolated feature processing in existing technologies, organically combines dynamic trajectory features of skeletal motion with semantic features of behavior video, and can more accurately reflect children's interactive intentions and behavior states. For example, it can judge the standardization of children's actions through skeletal motion features and understand the purpose of children's actions through behavior semantic features, making the system's interpretation of children's behavior more in-depth and accurate.

[0061] The interaction sensitivity calculation module introduces a pre-trained dynamic weighting network model to assign attention to the fused feature set and generate interaction response sensitivity distributions corresponding to different teaching segments. This design changes the fixed sensitivity mode of existing systems, allowing the system to dynamically adjust response strategies based on children's real-time behavior features. In segments where children actively participate, the system can increase sensitivity to enhance interactivity; in segments where focused learning is required, the system can appropriately reduce sensitivity to reduce interference, making teaching interactions more in line with children's behavior rhythm and teaching scene needs.

[0062] The abnormal behavior tracing module analyzes skeletal motion timing data and behavior video streams jointly and combines interaction response sensitivity distributions to perform root cause analysis of abnormal behavior. This function addresses the problem of existing systems' inability to locate the source of abnormalities and can accurately identify the teaching segment and behavior deviation type where abnormal behavior occurs, such as attention dispersion and action errors. By identifying the root cause of abnormalities, the system provides a clear direction for subsequent teaching adjustments, making teaching interventions more targeted.

[0063] The interaction strategy generation module generates an adaptive teaching strategy according to the abnormality tracing result and sends the adaptive teaching strategy to the robot motion control unit, so that the teaching behavior of the robot can respond to the state change of the children in real time. This dynamic adjustment mechanism changes the fixed teaching process mode of the existing system, can adjust the teaching content, rhythm or interaction mode in time according to the behavior abnormality of the children, meets the learning characteristics and needs of different children, makes the teaching process more flexible and efficient, and improves the participation and experience of the children in the teaching activity. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 A working principle diagram of the somatosensory interaction-based robot infant teaching system is provided.

[0065] Figure 2 A flowchart of the action semantic analysis processing of the interactive behavior video stream is provided.

[0066] Figure 3 A process diagram of the action semantic analysis processing of the interactive behavior video stream is provided.

[0067] Figure 4 A flowchart of the dynamic trajectory feature extraction processing of the skeletal point motion time sequence data is provided.

[0068] Figure 5 A process diagram of the dynamic trajectory feature extraction processing of the skeletal point motion time sequence data is provided.

[0069] Figure 6 A flowchart of the multi-objective optimization processing of the effective interaction strategy set is provided. DETAILED DESCRIPTION

[0070] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0071] Please refer to Figure 1 The present application provides a somatosensory interaction-based robot infant teaching system and device, and the method comprises:

[0072] The skeletal point motion time series data and interactive behavior video stream of the target infant are collected in real time by a depth camera and an inertial sensor. The somatosensory data acquisition module adopts a multi-sensor synchronization technology to ensure that the timestamps of the skeletal point motion data and the video stream are accurately aligned. The multi-modal feature fusion module extracts dynamic trajectory features from the skeletal point motion time series data to generate a skeletal motion feature vector; at the same time, it performs action semantic analysis on the interactive behavior video stream to generate a behavior semantic feature vector. Through cross-modal association fusion technology, the skeletal motion feature vector and the behavior semantic feature vector are fused into a unified fusion interaction feature set. The interaction sensitivity calculation module uses a pre-trained dynamic weighting network model to perform attention allocation on the fusion interaction feature set to generate a set of teaching interaction sensitivity coefficients, which reflects the interaction response sensitivity distribution of different teaching links. The abnormal behavior tracing module performs joint root cause analysis on the skeletal point motion time series data and the interactive behavior video stream based on the interaction response sensitivity distribution to identify the teaching link and the behavior deviation type of the abnormal source. The interaction strategy generation module generates an adaptive teaching strategy according to the abnormal source tracing result, and executes the strategy adjustment through the robot motion control unit to realize the dynamic optimization of the teaching process.

[0073] Embodiment 1: refer to Figure 2 The action semantic analysis processing of the interactive behavior video stream is a key link for the system to understand the behavior of infants. The collected interactive behavior video stream is segmented into action sequences, and a sliding window technology is used to divide the continuous video stream into multiple action segments with a fixed time span. The time length of each action segment is adaptively adjusted according to the dynamic characteristics of the teaching scene to adapt to the behavior rhythm changes of different teaching links. The segmented action segments maintain continuity and integrity in time to avoid breaking the action semantics caused by segmentation.

[0074] A pre-trained three-dimensional convolutional network model encodes the spatio-temporal features of each action segment. This model uses a multi-layer three-dimensional convolutional structure that can capture feature changes in both spatial and temporal dimensions of video data. In the spatial dimension, the model extracts local features of human posture and action form through convolution kernels; in the temporal dimension, the model captures the dynamic evolution process of actions using temporal convolution. After multiple layers of feature extraction, the model outputs an initial action feature vector that comprehensively represents the spatio-temporal characteristics of the action segment. The teaching behavior feature enhancement processing further optimizes the semantic expression ability of the initial action feature vector. The system retrieves a set of standard behavior templates associated with the current action segment from a pre-set infant behavior knowledge base. The knowledge base stores standard behavior sequences in typical teaching scenarios, including teacher demonstration actions, infant imitation actions, and interactive behavior patterns. A feature encoder performs semantic embedding processing on the set of standard behavior templates to generate a set of template feature vectors. This encoder uses a bidirectional long short-term memory network structure that can fully model the temporal dependency of standard behavior templates while preserving their semantic information.

[0075] In the feature fusion stage, the system calculates the similarity of the initial action feature vector and each template feature vector. The similarity calculation uses the cosine similarity measure, reflecting the closeness of the current action segment and the standard behavior template in the feature space. According to the similarity weight, the system performs weighted summation on the template feature vectors to generate the enhanced action feature vector. This process enables the initial action feature vector to fuse the semantic information of the standard behavior template, thereby improving the accuracy and robustness of the action representation.

[0076] The system inputs the enhanced action feature vectors of each action segment into the temporal position encoder for integration processing. The temporal position encoder uses the self-attention mechanism to dynamically model the temporal dependency between action segments. Through position encoding technology, the system preserves the temporal order information of the action segments while capturing long-range dependencies. The integrated behavior semantic feature vector not only contains the local features of individual action segments, but also covers the global action semantic information of the entire video stream.

[0077] In the entire action semantic analysis process, the system gradually improves the semantic richness of the behavior representation through multi-level feature extraction and fusion technology. The three-dimensional convolutional network model is responsible for capturing the bottom-level spatio-temporal features of the action, while the teaching behavior feature enhancement process introduces domain knowledge to optimize the semantic expression of the features. The temporal position encoder further integrates the global temporal information to ensure that the behavior semantic feature vector can fully reflect the behavior patterns of children in the teaching interaction. The system's action semantic analysis process has strong adaptability and can handle the behavior diversity in different teaching scenarios. By dynamically adjusting the segmentation length of the action segments and the feature fusion weight, the system can flexibly adapt to fast-paced or slow-paced teaching activities.

[0078] Referring to Figure 3 , the entire process of action semantic analysis processing of interactive behavior video stream is shown. The chart contains five subgraphs: action sequence segmentation processing: shows the process of dividing the continuous video stream into multiple fixed time span action segments, different gray bars represent the feature intensity of different segments; three-dimensional convolutional spatio-temporal feature encoding: the feature distribution of each action segment in the spatio-temporal dimension is shown through the gray heat map, the deeper the color, the higher the feature value; standard behavior template feature: shows the standard behavior template feature vector retrieved from the children's behavior knowledge base, different gray lines represent different templates; action segment and template similarity: uses a gray bar chart to show the similarity of the current action segment and each standard template, the deeper the bar color, the higher the similarity; enhanced behavior semantic feature vector: the enhanced feature after template weighted fusion is shown through the gray heat map, reflecting the feature representation after fusing the standard behavior semantic information.

[0079] Example 2: Referring to Figure 4, the dynamic trajectory feature extraction process of skeletal point motion time series data is an important link for the system to understand the movement behavior of young children. This process starts from the original skeletal point data obtained from depth cameras and inertial sensors, and gradually builds a high-dimensional representation that can fully reflect the movement characteristics of young children through multi-stage analysis. The system first detects abnormal fluctuations in the joint angle sequence by using a dynamic threshold method based on statistical distribution. This detection can adapt to the movement habits and amplitude differences of different children. After detecting abnormal fluctuation time segments, the system focuses on analyzing these intervals and uses a multi-scale sliding window technique to extract local motion trajectory data. The window size is automatically adjusted according to the joint motion amplitude, and a wider window is used for large-scale motion to capture the overall trend, and a narrower window is used for fine movements to retain detailed features.

[0080] The feature extraction of local trajectory data segments is realized by a pre-trained graph convolution network model. This model models the human skeletal structure as graph data, where the joints are the vertices of the graph and the skeletal connection relationships are the edges. Graph convolution operation can effectively capture the motion correlation between joints by propagating information on the topological structure of the graph. The model uses a multi-layer graph convolution structure, with shallow networks extracting the coordinated motion patterns of local joint groups and deep networks modeling the global coordination relationship of the whole body joints. After each local trajectory data segment is processed by the graph convolution network, the corresponding local motion feature vector is generated. These vectors contain not only the motion characteristics of the joints themselves, but also the dynamic correlation between the joints.

[0081] The global trajectory aggregation stage integrates scattered local motion features into a unified skeletal motion feature vector. This process uses an attention mechanism to dynamically calculate the contribution weight of each local feature, which depends on the time position and motion amplitude of the local trajectory data segment. Local features at key time points or containing significant motion changes will be given higher weights, so as to be highlighted in the global feature. Through this weighted fusion method, the skeletal motion feature vector generated by the system can take into account both the overall trend and key details of the motion process, forming a comprehensive portrayal of the movement behavior of young children.

[0082] In the cross-modal correlation fusion link, the system needs to effectively combine the skeletal motion features with the behavior semantic features. Since the two types of features may come from devices with different sampling frequencies, the system first performs time dimension alignment processing on the behavior semantic feature vector. This processing uses the dynamic time warping algorithm to map the behavior semantic features to the same time reference as the skeletal motion features while maintaining the temporal logic of the action unchanged. The aligned two types of features are sent to the skeletal behavior cross-attention mechanism, which establishes the mapping relationship between the skeletal motion trajectory and the behavior semantics by calculating the correlation matrix between the features.

[0083] The cross-attention mechanism adopts a bidirectional interaction architecture to simultaneously calculate the attention distribution of the skeletal motion features to the behavior semantic features and the attention distribution of the behavior semantic features to the skeletal motion features. This bidirectional design enables the two types of features to guide each other, and the feature extraction of the skeletal motion trajectory can refer to the context information of the behavior semantics, and the understanding of the behavior semantics can also combine the physical characteristics of the skeletal motion. Based on the calculated cross-modal correlation matrix, the system performs interactive reconstruction on the original features to generate interactive trajectory feature vectors and interactive semantic feature vectors that contain both skeletal motion details and fused behavior semantics.

[0084] The gated feature fusion is the last link of the cross-modal combination, and this process dynamically adjusts the fusion proportion of the two types of interactive features through a learnable gating mechanism. The gating signal is generated according to the context information of the current teaching scene, and for tasks that require accurate motion analysis, the skeletal motion features are preferred, while for tasks that require semantic understanding, the behavior semantic features are emphasized. The fused feature set not only retains the core information of the two types of original data, but also generates new joint representations through cross-modal interaction, which can more comprehensively reflect the comprehensive state of children in teaching interaction.

[0085] In terms of technical implementation, the dynamic trajectory feature extraction and cross-modal fusion module of the system adopts a distributed computing architecture. The computing tasks of the graph convolution network model and the attention mechanism are distributed to multiple processing units for parallel execution, significantly improving the processing efficiency. In view of the high-dimensional characteristics of the skeletal point data, the system adopts feature dimension reduction and sparsification technology to reduce the computational complexity while retaining key information. The time alignment and attention calculation in the cross-modal fusion process are also optimized by algorithm to ensure that the analysis accuracy remains high under the real-time requirement.

[0086] The anti-interference ability of the system is fully embodied in this embodiment. For skeletal point data anomalies caused by sensor noise or temporary occlusion, the dynamic trajectory feature extraction module repairs the data through temporal context analysis and spatial consistency verification. During the cross-modal fusion process, when the quality of a certain modality data decreases, the system will automatically adjust the fusion weight and rely more on the reliable information of another modality.

[0087] Referring to Figure 5, the dynamic trajectory feature extraction process of the skeletal point motion time series data is shown. The chart contains six subgraphs: original skeletal point motion time series data: the motion trajectory of multiple joint points is shown, and different gray lines represent different joint motion patterns; abnormal action interval detection: it shows how the system identifies abnormal fluctuation time segments in the joint angle sequence, and the gray filled area identifies the detected abnormal interval; multi-scale sliding window sampling processing: different gray lines are used to show the abnormal indicators after moving average processing under different window sizes, which are used to capture motion features of different time scales; joint correlation feature matrix: the motion correlation between joints is shown through a gray matrix, and the deeper the color, the higher the correlation; local motion feature vector: it shows the local motion features extracted from different time segments, and different gray lines represent feature vectors of different time periods; global skeletal motion feature vector: the global motion features aggregated by the attention mechanism are shown using a gray bar chart, and the deeper the bar color, the more significant the feature dimension.

[0088] Example 3: The processing process of the dynamic weighted network model for the fusion interaction feature set is the core link of the system to realize the teaching interaction sensitivity analysis. The model adopts a hierarchical processing architecture, which gradually extracts key information from the original fusion features and establishes the correlation between the teaching links and the interaction responses. The system first inputs the fusion interaction feature set into the feature screening layer, which uses sparse coding principle to compress and select high-dimensional features. In the sparse coding process, each input feature is represented as a linear combination of a small number of basis vectors, which are obtained by pre-training and can capture typical patterns in teaching interaction. The core interaction feature set output by the feature screening layer retains the most discriminative information in the original data, while significantly reducing the feature dimension, providing more compact data representation for subsequent processing.

[0089] The cross-correlation layer performs high-order feature combination analysis on the core interaction feature set. This layer uses tensor operations to mine the potential correlations between different teaching links and expands one-dimensional feature vectors into multi-dimensional correlation feature tensors. In the tensor construction process, the system uses the following calculation method:

[0090]

[0091] Among them: represents an element in the three-dimensional correlation feature tensor, , and represent the core interaction feature matrices from different feature channels, represents the tensor product operation, is a learnable weight coefficient, is the number of feature channels. This calculation process can capture the complex multi-factor correlations in teaching interactions, such as the nonlinear relationship between the attention changes of young children and the difficulty of teaching materials. The generated correlation feature tensor contains rich cross-feature information, providing a multi-dimensional data basis for sensitivity analysis.

[0092] The sensitivity distribution layer calculates the interaction response weights of different teaching segments based on the correlation feature tensor. This layer adopts a multi-branch neural network structure, with each branch corresponding to a teaching segment type. The network first separates the input tensor by channel, mapping features of different dimensions to the corresponding teaching segment analysis space. Within each branch, segment-specific features are extracted through cascaded convolution and nonlinear transformation, and finally the interaction response sensitivity value of the segment is generated through normalization processing. The calculation of sensitivity values considers factors such as historical interaction patterns, current teaching status, and individual differences of young children, forming a dynamically weighted evaluation result.

[0093] The feature weighting aggregation stage combines the correlation feature tensor with the sensitivity distribution to generate the final set of teaching interaction sensitivity coefficients. The aggregation process uses an attention-based feature selection mechanism, with high-sensitivity feature dimensions being assigned greater aggregation weights. This dynamic weighting method ensures that the system can flexibly adjust the attention to different interaction features according to actual teaching needs. For example, in teaching segments that require fine motor guidance, the system will enhance the sensitivity to skeletal movement features; while in segments that emphasize social interaction, it will increase the response weight of behavior semantic features.

[0094] The training process of the dynamic weighting network model uses an end-to-end optimization strategy. The loss function is designed as a combination of feature reconstruction error and sensitivity prediction error, where the feature reconstruction error measures the reconstruction quality of core interaction features, and the sensitivity prediction error evaluates the accuracy of teaching segment classification. The training data contains interaction records in various typical teaching scenarios, covering behavior patterns of young children of different ages and teaching interaction types. Through large-scale data training, the model can learn feature representations and sensitivity evaluation criteria with generalization ability.

[0095] In terms of model architecture design, the dynamic weighting network adopts a modular component structure. The feature selection layer, cross-correlation layer, and sensitivity distribution layer can be independently updated and replaced, facilitating customized adjustments for specific teaching needs. The model also supports incremental learning mechanisms, allowing it to adapt to new teaching scenarios and groups of young children through local parameter updates without retraining the entire network. This flexibility enables the system to continuously optimize its interaction analysis capabilities, maintaining long-term teaching adaptability.

[0096] The real-time processing capability of the system is optimized in this embodiment. The dynamic weighted network model adopts a lightweight design, reducing computational complexity through depthwise separable convolution and parameter sharing techniques. The construction process of the correlation feature tensor utilizes tensor decomposition techniques to convert high-dimensional operations into a combination of multiple low-dimensional operations, significantly improving processing efficiency. The multi-branch structure of the sensitivity allocation layer supports parallel computing, and different teaching stages can be analyzed simultaneously. These optimization measures ensure that the system can complete complex sensitivity analysis in real time during the teaching process, providing support for immediate strategy adjustment.

[0097] The dynamic weighted network model exhibits strong robustness in handling noise and uncertainty. In response to potential disturbances during sensor data collection, the model sets up an outlier suppression mechanism in the feature screening layer, identifying and removing unreliable feature components through statistical testing. The tensor operation of the cross-correlation layer has inherent noise smoothing properties, which can offset the influence of random errors to some extent. The sensitivity allocation layer uses an integrated decision-making strategy to generate the final sensitivity evaluation by combining the analysis results of multiple feature channels, reducing the disturbance of single feature fluctuations on the overall judgment.

[0098] The model's explainability design is also an important feature of this embodiment. The dynamic weighted network not only generates sensitivity coefficients but also provides feature contribution analysis, intuitively showing the impact of each factor on the final decision. This transparent analysis process helps teachers understand the system's decision-making logic and facilitates the smooth development of human-machine collaborative teaching. The system also supports the visualization of sensitivity analysis results, displaying response mode changes in different teaching stages through a graphical interface, providing intuitive references for teaching reflection and improvement.

[0099] The application scope of the dynamic weighted network model is not limited to basic teaching interaction analysis. By adjusting the input feature types and output sensitivity definitions, the model can be extended to a wider range of educational scenarios. For example, in the field of special education, the model can be configured to focus on interaction features of specific behavior disorders; in language teaching, it can be adjusted to emphasize voice and expression sensitivity analysis. This scalability enables the system to adapt to diverse educational needs and provide customized interaction analysis solutions for different scenarios.

[0100] In the actual operation of the robot infant teaching system, the joint root cause analysis module is responsible for accurately tracing the abnormal teaching behavior. Taking a specific building block teaching scenario as an example, the system detects that the infant has failed multiple times when imitating the building of a tower structure, at which point the system starts the abnormal trace process. First, the system filters out abnormal items that exceed the preset dynamic threshold based on the interaction response sensitivity distribution, generating a candidate abnormal feature set. This set includes multiple dimensional abnormal indicators, such as abnormal hand joint movement amplitude, insufficient attention duration, etc.

[0101] Table 1: Candidate abnormal feature set in this scenario.

[0102]

[0103] In the reverse feature analysis stage, the system maps the abnormal features in the table to specific motion abnormality and behavior deviation descriptions. For the AF-102 feature, the system identifies that the child has difficulty in fine motor control, as evidenced by inaccurate finger pinch angles when picking up blocks. The AF-203 feature reflects the child's lack of attention persistence, as evidenced by frequent head turning that causes the line of sight to deviate from the teaching area. The AF-305 feature shows that the child has a deviation in spatial position judgment, resulting in inaccurate block placement.

[0104] The causal relationship between these abnormal features is analyzed by the traceability reasoning model using a graph structure. The model constructs a reasoning graph containing three layers of nodes: the bottom layer nodes represent specific abnormal features, the middle layer nodes represent potential teaching session problems, and the top layer nodes correspond to root cause categories. Taking the AF-102 feature as an example, the model finds that this feature is strongly associated with the two middle layer nodes "insufficient fine motor guidance" and "teaching pace too fast" by analyzing historical data. Combined with the AF-203 attention dispersion feature, the model further infers that the root cause may be "teaching pace mismatching the child's receptive ability".

[0105] In the confidence evaluation stage, the system calculates the support degree index of each traceability path. The support degree calculation considers three dimensions: historical case matching degree, logical coherence between features, and real-time situation conformity. In the above block building case, the "teaching pace mismatching" path obtains the highest comprehensive score because it explains both hand movement abnormalities and attention dispersion features, and is highly consistent with the current teaching progress data. The system finally determines this path as the target traceability result, clearly indicating that the abnormality originates from the rhythm control problem of the collective teaching session, and the behavior deviation type is motion execution difficulty accompanied by attention dispersion.

[0106] The adaptive teaching strategy generation process takes targeted adjustments based on the traceability result. The system first queries the historical strategy library to retrieve a set of historical strategies related to "teaching pace adjustment". This set contains various intervention methods, such as segmented demonstration, pace slowing, and individual guidance, etc. In the effectiveness screening stage, the system preferentially selects strategies that have improved both motion execution and attention maintenance in similar past cases. For example, the "step-by-step decomposition + voice prompt" combined strategy in the strategy library shows good comprehensive effects in historical records.

[0107] In the strategy optimization phase, the system conducts multi-dimensional evaluation on the filtered effective strategies. Taking the building block scenario as an example, the system weighs three key indicators: task completion progress, action accuracy improvement, and attention maintenance duration. Step-by-step demonstration strategies may improve action accuracy but may prolong task time; while increasing voice prompts can help maintain attention but may increase information load. The system balances these factors to ultimately generate an adaptive strategy that divides the building steps into three stages, each accompanied by concise voice prompts.

[0108] During strategy execution, the system continuously monitors the child's response changes. For the building block scenario, the adjusted teaching strategy is as follows: the robot first demonstrates the base building, waits for the child to complete it, then demonstrates the middle layer, and finally guides the top building. At the beginning of each stage, a short prompt voice is played, such as "Now please place the blocks at the marked position." The system verifies the strategy execution effect by real-time collection of the child's joint motion data and gaze direction. If it is found that attention is again dispersed or action error rate rises, the system will trigger a second adjustment of the strategy.

[0109] The closed-loop process of abnormality tracing and strategy adjustment reflects the dynamic adaptation capability of the system. In the subsequent stage of the building block scenario, the system finds that the child has new difficulties after completing the base. Tracing analysis shows that this time it is due to insufficient spatial positioning ability leading to the middle layer block alignment problem. The system immediately increases the visual marker auxiliary strategy under the maintenance of the step-by-step teaching framework, displaying alignment reference lines on the block contact surface. This layer-by-layer deepening adjustment method enables the teaching strategy to dynamically evolve with the child's learning process.

[0110] The knowledge updating mechanism of the tracing reasoning model is also an important feature of this embodiment. The system regularly incorporates new teaching cases and strategy effect feedback into the analysis model, optimizing the reasoning rules through incremental learning. For example, when it is found that a certain type of attention dispersion is associated with changes in the classroom environment, the model will automatically adjust the weights of related features. This continuous evolution capability enables the system's analysis accuracy to continuously improve over time, gradually forming a customized analysis mode for specific teaching environments and groups of children.

[0111] Embodiment 5: see Figure 6, the process of multi-objective optimization for generating adaptive teaching strategies is the key step for the system to realize personalized teaching. Based on the effective interaction strategy set screened in the previous stage, the system determines the adjustment scheme most suitable for the current teaching scene and the state of the children through multi-dimensional comprehensive evaluation and optimization. The system first constructs a three-dimensional optimization target space, with each dimension representing a core index that the teaching strategy needs to consider. The first dimension is the teaching progress completion rate, reflecting the effect of the strategy on the advancement of the established teaching goal; the second dimension is the behavior deviation correction rate, measuring the improvement of the strategy on abnormal behavior; the third dimension is the interaction response timeliness, evaluating the timeliness and fluency of the strategy execution. These three dimensions together form a three-dimensional framework for strategy evaluation, ensuring that the generated strategy can balance teaching effectiveness and interaction experience.

[0112] In the target space mapping stage, the system converts each candidate strategy into a coordinate point in the three-dimensional space. The calculation of coordinate values is based on the historical execution data of the strategy and the real-time characteristics of the current scene. The teaching progress completion rate is obtained by analyzing the task advancement speed of the strategy in similar scenes, while considering the remaining task amount of the current teaching stage; the behavior deviation correction rate is comprehensively evaluated in combination with the real-time feedback data of the children and the past correction effect of the strategy; the interaction response timeliness is calculated according to the preparation time and implementation duration required for the strategy execution. This mapping method enables the abstract teaching strategy to be converted into a quantifiable and comparable spatial position, providing a direct basis for subsequent optimization selection.

[0113] The multi-objective optimization algorithm finds the optimal strategy set in the three-dimensional target space. The system uses an improved Pareto optimization method to select candidate schemes that are not completely dominated by other strategies in all three dimensions. These schemes are distributed on the Pareto front, representing different emphasis on strategy selection: some focus on quickly advancing the teaching progress, some focus on thoroughly correcting behavior deviation, and some focus on maintaining the fluency of interaction. The algorithm ensures that the strategy distribution on the front is representative by calculating the crowding degree, avoiding excessive concentration in a certain local area. This processing method preserves the diversity of strategies, providing sufficient selection space for subsequent context adaptation.

[0114] The strategy adaptation model adaptively scores the candidate strategies based on the real-time state of the current teaching scene. The scoring process considers three core factors: the immediate state characteristics of the children, the environmental conditions, and the requirements of the teaching stage. For the state of the children, the model analyzes their current attention level, emotional state, and fatigue level; environmental conditions include classroom noise level, light intensity, and the presence of interference factors; teaching stage requirements involve course progress arrangement and the core goal of the current teaching. The model dynamically adjusts the proportion of each factor through a hierarchical weight distribution mechanism. For example, it may pay more attention to teaching progress completion at the end of the course, while giving priority to the comfort of interaction when the children show obvious signs of fatigue.

[0115] The final strategy selection phase, the system comprehensively compares the adaptability scores of each candidate strategy. The strategy with the highest score may not necessarily perform optimally in all dimensions, but rather achieves the best balance under the current specific situation. Taking the block classification teaching as an example, the system may choose a strategy that is mediocre in progress promotion but can significantly improve the accuracy of children's operations, rather than simply pursuing a solution that quickly completes the task. This selection mechanism reflects the system's understanding of the nature of teaching, focusing not only on the completion of surface tasks, but also on the real gains of children in the process.

[0116] The fine-tuning of strategy parameters is the last link in the optimization process. The system adjusts the execution details of the selected strategy based on the scoring results, including action demonstration speed, language prompt frequency, and response waiting time. These parameter adjustments are based on real-time feedback data from children and gradually optimized in small steps. For example, when the system detects that children have difficulty understanding the current demonstration speed, it will automatically slow down the robot's motion amplitude and increase the stage pause. This fine-tuned parameter adjustment allows the general strategy to adapt to individual differences and achieve truly personalized teaching.

[0117] The dynamic nature of multi-objective optimization processing is reflected in the continuous strategy evaluation mechanism. The system continues to monitor the implementation of the three target dimensions while executing the selected strategy. When the actual effect deviates from the expected, the system triggers the re-optimization process of the strategy. This closed-loop optimization mechanism ensures that the teaching strategy can dynamically adjust to changes in the teaching process and the state of children, avoiding the rigidity that may be caused by fixed strategies. In actual teaching, this dynamic nature is manifested in the gradual evolution of strategies, from behavior correction as the main focus in the early stage, to gradually transitioning to ability cultivation, and finally guiding autonomous exploration.

[0118] The optimization process of the system also considers the long-term impact of teaching strategies. In addition to the three immediate target dimensions, the model also implicitly assesses the potential impact of strategies on children's learning interest and self-confidence. By analyzing the long-term effects of similar strategies in historical data, the system will prefer strategies that not only solve the current problem, but also benefit future development. For example, in the selection of action guidance strategies, the system may prefer guided discovery rather than direct demonstration, as the former, although slower in initial progress, is more conducive to cultivating children's exploration ability and problem-solving skills.

[0119] The implementation of multi-objective optimization processing relies on a rich policy knowledge base constructed by the system. This knowledge base not only stores the basic descriptions and execution parameters of various policies, but also records a large amount of effect data of historical execution cases. Each policy entry is associated with multi-dimensional evaluation indicators and applicable condition descriptions, providing sufficient decision basis for optimization algorithms. The knowledge base organizes policy data in a graph structure, facilitating the system to quickly retrieve relevant policies and discover potential combination schemes. This knowledge representation supports innovative combination of policies, enabling the system to generate new solutions beyond the original policy library.

[0120] The optimization process design of the system also takes into account the teaching ethics. In the selection of strategies, the system will avoid those that may harm the autonomy of young children, although they can quickly produce results. For example, excessive prompting or substitute strategies will not be used to improve short-term task completion rates. This design concept makes the teaching behavior of the system consistent with the basic principles of child development, protecting the learning subjectivity and exploration pleasure of young children while pursuing teaching effectiveness.

[0121] The implementation of multi-objective optimization processing also relies on high-efficiency real-time computing capability. The system adopts a hierarchical computing architecture, distributing time-consuming policy evaluation tasks to multiple processing units for parallel execution. The core optimization algorithm is specially optimized to complete the screening and comparison of large-scale policy sets within a limited time. This computing efficiency guarantees the real-time response capability of the system in actual teaching, so that the complex multi-objective optimization process will not cause significant interaction delay.

[0122] Embodiment 6: Real-time feedback and strategy optimization module constitutes the core mechanism of the system to realize dynamic adjustment of teaching. This module continuously runs during the execution of adaptive teaching strategies by the robot, enabling the teaching strategies to evolve continuously following the real-time state of young children through multi-source data acquisition and incremental learning technology. The system collects physiological response data of young children through non-invasive sensing devices, including heart rate variability indicators measured by wrist-worn sensors and emotional arousal data obtained by finger contact skin conductance sensors. These physiological signals are recorded at millisecond-level time resolution, forming continuous data streams reflecting the internal state of young children. At the same time, multi-view visual sensors deployed in the classroom environment capture the behavior feedback data of young children, including facial micro-expression changes, body movement amplitude, and interaction modes with teaching materials. These multi-modal data streams are aligned through time synchronization technology, constructing a multi-dimensional observation system that comprehensively reflects the acceptance state of young children.

[0123] The analysis of physiological response data employs a time-series pattern-based analysis method. The system identifies the rhythmic variation characteristics in heart rate variability signals, distinguishing different state patterns such as attention concentration, cognitive load, and emotional fluctuations. Skin conductance signals are analyzed for their amplitude fluctuations and duration characteristics to assess the level of tension and engagement of young children when facing teaching tasks. The analysis results of these physiological indicators are associated with the execution time points of teaching strategies, forming the basis for internal evaluation of strategy effectiveness. For example, when the system detects a specific rhythmic enhancement in heart rate variability indicators after strategy adjustment, it may reflect an improvement in the cognitive engagement of young children; while a steady decline in skin conductance levels may indicate an improvement in emotional state.

[0124] The processing of behavioral feedback data integrates computer vision and motion analysis techniques. Facial expression recognition algorithms extract eye and mouth region motion unit features from video streams, distinguishing states such as confusion, understanding, and excitement. Body motion analysis focuses on changes in the way young children interact with teaching materials, such as grip strength adjustments during building with blocks, and changes in pen pressure during drawing. These external behavioral characteristics are cross-verified with internal physiological response data to form a multi-angle evaluation of strategy execution effectiveness. The system pays special attention to subtle changes in behavioral feedback, such as brief shifts in eye gaze direction or slight hesitation in finger movements, which often reflect key points that need to be optimized in teaching strategies.

[0125] The incremental learning model is responsible for converting real-time collected feedback data into strategy optimization knowledge. The model uses an online learning architecture, taking the currently executed teaching strategy as a benchmark, and calculates a strategy correction coefficient by analyzing the deviation direction and degree between feedback data and expected results. This coefficient is a multi-dimensional vector that contains adjustment suggestions for each parameter in the strategy, such as correction amount for action demonstration speed, change amplitude for language prompt frequency, and adjustment value for response waiting time. The calculation process of the coefficient considers the time-series characteristics and multi-modal consistency of feedback data. When physiological responses and behavioral feedback point to the same conclusion, the correction amplitude increases accordingly; when different data sources have differences, a conservative adjustment method is adopted.

[0126] Dynamic adjustment of strategy parameters follows the principle of gradual optimization. The system does not make significant strategy changes based on single feedback data, but rather forms a trend judgment through continuous observation, and then implements step-by-step adjustments. This design avoids strategy oscillation caused by accidental factors, ensuring the stability of the teaching process. During the adjustment process, the system maintains a strategy evolution trajectory record, tracking the historical change path of each parameter and the corresponding effect, forming a traceable optimization process. For example, in language prompt frequency adjustment, the system may start with a prompt every two minutes, and gradually fine-tune to the optimal interval based on the child's response, rather than directly using a pre-set fixed frequency.

[0127] The verification of the adjusted strategy employs a closed-loop testing method. After implementing parameter adjustments, the system sets an observation window period to focus on monitoring the changes in feedback indicators related to the adjustment targets. For example, when adjusting the demonstration speed of actions, the system will pay special attention to the accuracy and completion time of the children's action imitation; when changing the language prompt method, it will focus on analyzing the children's gaze following rate and response delay. This target-oriented verification mechanism ensures that each adjustment can obtain a clear effect evaluation, providing a reliable basis for subsequent optimization. During the verification process, the system will distinguish between short-term adaptive effects and long-term stable effects, avoiding misjudgment of the children's temporary adaptation to new strategies as strategy effectiveness.

[0128] The implementation effect of real-time feedback and strategy optimization is reflected in the dynamic adaptability of the teaching process. In the block classification teaching case, the system initially adopts the standard strategy of classifying by shape. Through real-time monitoring, it is found that the children have increased skin conductance levels and action hesitation when classifying round blocks. The system gradually adjusts the strategy: first, increase the tactile prompt of round blocks, then introduce size contrast to assist classification, and finally adjust to classifying by shape-size composite characteristics. This progressive strategy evolution keeps the teaching difficulty within the optimal learning interval of the children, neither too simple to lose challenge nor too difficult to cause frustration.

[0129] The incremental learning mechanism of the system has the characteristics of knowledge accumulation. The optimization experience generated by each teaching interaction is abstracted as rule knowledge and stored in the strategy optimization knowledge base. These knowledge are indexed according to dimensions such as teaching scene type, children's age stage, and behavior characteristics, forming transferable optimization experience. When encountering similar teaching situations, the system can quickly retrieve relevant optimization history to speed up the strategy adjustment process in new scenarios. The knowledge base also records long-term effect data of different strategy adjustment schemes, enabling the system to evaluate the comprehensive benefits of strategy optimization from a more macro time scale, avoiding short-sighted local optimization.

[0130] The timeliness of real-time feedback processing is an important technical feature of this embodiment. The system uses a streaming computing architecture to control the whole process of data collection, analysis, and strategy adjustment within seconds. This real-time nature ensures the natural flow of teaching interaction and avoids interaction disconnection caused by system delay. The data processing pipeline uses a priority scheduling mechanism to ensure that critical feedback signals can be responded to immediately, while secondary indicators can be adjusted with a slightly looser time requirement. For example, the children's puzzled expression or sudden action pause will trigger immediate interrupt processing, while the physiological indicator fluctuations under normal conditions will be included in the periodic adjustment plan.

[0131] Multi-modal fusion analysis of the feedback data is another significant feature of the present embodiment. Instead of relying on a single data source to make a judgment, the system integrates the internal state indication of physiological signals and the external evidence of behavioral performance to form a stereoscopic evaluation conclusion. When the facial expression shows confusion but the body movement remains smooth, the system may judge that the child is on the edge of a cognitive breakthrough and choose to maintain the current strategy for observation; while when the physiological indicators show tension accompanied by behavioral retreat, it will immediately start the adjustment of the supportive strategy. This multi-angle cross-verification analysis method greatly improves the accuracy of state judgment, making the strategy adjustment more accurate and effective.

[0132] The optimization process of the system also takes into account the individual differences and state fluctuations of the children. The same feedback data may correspond to different optimization directions at different children or different time points. By establishing a child's personal baseline profile, the system distinguishes between their normal performance and temporary changes, avoiding misjudgment of individual characteristics as problems that need to be corrected. For example, for children with a naturally active nature, a higher movement frequency may not require intervention; while the same performance in a usually quiet child may need to be concerned about the underlying reasons. This personalized interpretation ability enables the system to go beyond mechanical standardized responses and achieve truly individualized teaching support.

[0133] It should be noted that the relational terms herein such as first and second and the like are used solely to distinguish one entity or action from another, without necessarily requiring or implying any such actual relationship or order between or among such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0134] Although embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, replacements and changes can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A robot-based early childhood education system based on motion-sensing interaction, characterized in that, The system includes: The motion data acquisition module is used to acquire the time-series data of the skeletal movement of the target child and the video stream of the interactive behavior through a depth camera and an inertial sensor; The multimodal feature fusion module is used to perform dynamic trajectory feature extraction processing on the motion time series data of the skeleton points to generate skeleton motion feature vectors, perform action semantic parsing processing on the interactive behavior video stream to generate behavior semantic feature vectors, and perform cross-modal association and fusion of the skeleton motion feature vectors and the behavior semantic feature vectors to generate a fused interactive feature set. The interaction sensitivity calculation module is used to call a pre-trained dynamic weighted network model to perform attention allocation processing on the fused interaction feature set and generate a set of teaching interaction sensitivity coefficients, which includes the distribution of interaction response sensitivity corresponding to different teaching stages. An abnormal behavior tracing module is used to perform joint root cause analysis on the motion time series data of the skeletal points and the video stream of the interactive behavior based on the interaction response sensitivity distribution, and generate teaching behavior abnormality tracing results. The teaching behavior abnormality tracing results are used to indicate the teaching link and behavior deviation type from which the abnormality originates. An interaction strategy generation module is used to generate an adaptive teaching strategy based on the abnormal teaching behavior tracing results, and send the adaptive teaching strategy to the robot motion control unit; The process of calling a pre-trained dynamically weighted network model to perform attention allocation on the fused interactive feature set, generating a set of teaching interaction sensitivity coefficients, including: The fused interactive feature set is input into the feature filtering layer of the dynamic weighted network model for key feature extraction, resulting in a filtered core interactive feature set. The cross-association layer of the dynamic weighted network model is then used to perform high-order feature combination processing on the core interactive feature set, generating an associated feature tensor. The sensitivity allocation layer of the dynamic weighted network model is used to calculate the weights of different teaching stages on the associated feature tensor, generating an interactive response sensitivity distribution for different teaching stages. Based on the interactive response sensitivity distribution, the associated feature tensor is subjected to feature weighting and aggregation processing to generate the teaching interaction sensitivity coefficient set. The sensitivity allocation layer adopts a multi-branch neural network structure, with each branch corresponding to a type of teaching segment. Within each branch, segment-specific features are extracted through cascaded convolutions and nonlinear transformations, and the interaction response sensitivity value for that segment is generated through normalization. The calculation of the sensitivity value comprehensively considers historical interaction patterns, current teaching status, and individual differences among children, forming a dynamically weighted evaluation result.

2. The robot-based early childhood education system based on motion-sensing interaction according to claim 1, characterized in that, The step of performing action semantic parsing processing on the interactive behavior video stream to generate a behavior semantic feature vector includes: The interactive video stream is processed by action sequence segmentation to obtain multiple continuous action segments; A pre-trained 3D convolutional network model is invoked to perform spatiotemporal feature encoding on each of the continuous action segments to generate an initial action feature vector; The initial action feature vector is subjected to teaching behavior feature enhancement processing, including: matching a set of standard behavior templates associated with the current action segment from a preset children's behavior knowledge base; inputting the set of standard behavior templates into a feature encoder for template semantic embedding processing to generate a set of template feature vectors; and performing similarity weighted fusion between the set of template feature vectors and the initial action feature vector to generate an enhanced action feature vector. The enhanced action feature vectors of each continuous action segment are integrated by temporal position encoding to generate the behavioral semantic feature vector.

3. The robot-based early childhood education system based on motion-sensing interaction according to claim 1, characterized in that, The step of performing dynamic trajectory feature extraction processing on the temporal data of the skeletal point motion to generate a skeletal motion feature vector includes: The abnormal motion interval detection process is performed on the motion time sequence data of the skeletal points to identify abnormal fluctuation time segments in the joint angle change sequence; Within the abnormal fluctuation time segment, the original skeletal data is subjected to multi-scale sliding window sampling to obtain multiple local trajectory data segments; A pre-trained graph convolutional network model is invoked to perform joint association feature extraction processing on each of the local trajectory data segments, generating a local motion feature vector; Global trajectory aggregation processing is performed on the local motion feature vectors of each of the local trajectory data segments to generate the skeletal motion feature vector.

4. The robot-based early childhood education system based on motion-sensing interaction according to claim 1, characterized in that, The step of cross-modal association and fusion of the skeletal motion feature vector and the behavioral semantic feature vector to generate a fused interactive feature set includes: The behavioral semantic feature vector is aligned to the time dimension and mapped to the same timestamp sequence as the skeletal motion feature vector. A cross-attention mechanism for skeletal behavior is constructed, and the cross-modal correlation matrix between the trajectory features of each timestamp in the skeletal motion feature vector and the semantic features of the corresponding timestamp in the behavior semantic feature vector is calculated. Based on the cross-modal correlation matrix, the skeletal motion feature vector and behavioral semantic feature vector are subjected to bidirectional feature interaction processing to generate interactive trajectory feature vector and interactive semantic feature vector. The interaction trajectory feature vector and the interaction semantic feature vector are subjected to gated feature fusion processing to generate the fused interaction feature set.

5. The robot-based early childhood education system based on motion-sensing interaction according to claim 1, characterized in that, The method of performing joint root cause analysis on the skeletal point motion time-series data and interactive behavior video stream based on the interactive response sensitivity distribution to generate abnormal teaching behavior tracing results includes: Based on the interaction response sensitivity distribution, abnormal sensitivity items exceeding a preset threshold are filtered to generate a candidate abnormal feature set; Each abnormality sensitivity item in the candidate abnormality feature set is subjected to reverse feature parsing to determine its corresponding skeletal motion abnormality description and behavioral semantic deviation description. The pre-trained source tracing inference model is invoked to perform joint causal analysis on the description of abnormal skeletal movement and the description of deviation in behavioral semantics, generating a set of abnormal source tracing paths; The set of abnormal tracing paths is subjected to confidence evaluation, and the target tracing path with the highest confidence is selected as the abnormal tracing result of the teaching behavior.

6. The robot-based early childhood education system based on motion-sensing interaction according to claim 5, characterized in that, The step of generating an adaptive teaching strategy based on the abnormal teaching behavior tracing results includes: The analysis reveals the abnormal teaching processes and behavioral deviation types in the results of the abnormal teaching behavior tracing. Match a set of historical adjustment strategies associated with the teaching segment from the history teaching strategy library that corresponds to the source of the anomaly; Based on the behavioral deviation type, the set of historical adjustment strategies is filtered for effectiveness to obtain a set of effective interaction strategies. The effective set of interaction strategies is subjected to multi-objective optimization to generate adaptive teaching strategies that meet the teaching constraints.

7. The robot-based early childhood education system based on motion-sensing interaction according to claim 6, characterized in that, The step of performing multi-objective optimization on the set of effective interaction strategies to generate adaptive teaching strategies that meet teaching constraints includes: Construct a three-dimensional optimization target space that includes teaching progress completion rate, behavioral deviation correction rate, and interaction response timeliness; Map each effective interaction strategy to a coordinate point in the three-dimensional optimization target space; A set of candidate policies located on the Pareto front is selected using a multi-objective optimization algorithm; The strategy adaptation model is invoked to perform adaptive scoring on the candidate strategy set based on the real-time status of the current teaching scenario; The candidate strategy with the highest adaptability score is selected as the adaptive teaching strategy.

8. The robot-based early childhood education system based on motion-sensing interaction according to claim 1, characterized in that, Also includes: The real-time feedback module is used to continuously collect physiological response data and behavioral feedback data of children during the process of the robot executing the adaptive teaching strategy. The strategy optimization module is used to input the physiological response data and behavioral feedback data into the incremental learning model for dynamic strategy optimization processing, and generate an optimized adaptive teaching strategy.

9. The robot-based early childhood education system based on motion-sensing interaction according to claim 8, characterized in that, When the strategy optimization module performs dynamic strategy optimization processing, it includes: The data on the effectiveness of the current adaptive teaching strategy are correlated with the physiological response data to generate a strategy correction coefficient. The interactive response parameters in the adaptive teaching strategy are dynamically adjusted based on the strategy correction coefficient. The adjusted interactive response parameters are matched and verified with the behavioral feedback data to generate the optimized adaptive teaching strategy.

Images