A learner intention reasoning method and system for an online learning scenario

Abstract

The application provides a learner intention reasoning method and system for an online learning scene, comprising: acquiring an online learner image, synchronously performing head posture recognition, trunk posture recognition and interactive behavior recognition through a multi-task joint training framework to obtain enhanced shared features; constructing a heterogeneous graph containing an expression node, a posture node and an object node from the obtained enhanced shared features; based on the constructed heterogeneous graph, performing feature coding through hierarchical feature extraction to output a reasoning intention. Through the weighted sum of two-level label comparison loss, the optimization objectives of different levels of features are taken into account, so that more fine semantic expression and higher intention reasoning accuracy are realized in a complex online learning scene.

Description

Technical Field

[0001] This invention belongs to the field of behavioral intention reasoning technology, and specifically relates to a learner intention reasoning method and system for online learning scenarios. Background Technology

[0002] With the rapid development of online education, accurately identifying learners' behavioral states and inferring their learning intentions has become a key technology for improving teaching quality.

[0003] Compared to the multi-person, multi-object scenarios in traditional offline classrooms, online learning images typically depict single-person, multi-object scenarios and contain richer, finer-grained information, which helps to achieve more refined semantic expression and higher accuracy in intent reasoning.

[0004] Existing research largely focuses on fields such as autonomous driving, lacking dedicated methods for online learning. Single-task learning of state recognition ignores the inherent connections between facial expressions, postures, and interactive behaviors, leading to limited model performance. Online learning of intent exhibits hierarchical characteristics, posing challenges to the semantic representation of intent and resulting in poor performance on imbalanced datasets and small sample datasets. Summary of the Invention

[0005] The purpose of this invention is to provide a learner intent reasoning method and system for online learning scenarios to solve the above-mentioned problems.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a learner intent reasoning method for online learning scenarios, comprising:

[0008] Images of online learners are acquired, and head pose recognition, torso sitting posture recognition, and interactive behavior recognition are performed simultaneously through a multi-task joint training framework to obtain enhanced shared features.

[0009] The obtained enhanced shared features are used to construct a heterogeneous graph containing expression nodes, pose nodes, and object nodes;

[0010] Based on the constructed heterogeneous graph, hierarchical feature extraction is used for feature encoding to output the reasoning intent.

[0011] Furthermore, the acquisition of online learner images, and the simultaneous execution of head pose recognition, torso sitting posture recognition, and interactive behavior recognition through a multi-task joint training framework, includes:

[0012] The multi-task joint training framework consists of five stages: image input, feature extraction, feature fusion, task classification, and result output. In the feature extraction stage, the expression, pose, and interaction behavior sub-tasks are extracted using pre-trained InceptionResNet, OpenPose, and Detr expert networks, respectively. In the feature fusion stage, shared feature fusion is performed on the relevant features. In the task classification stage, different classification heads are used to learn the classification of the relevant task features fused with shared features, and the multi-task loss is calculated. Finally, the recognition results of each task are output.

[0013] The feature classification stage classifier consists of a fully connected layer and a Softmax function, which effectively distinguishes different categories of facial expressions and poses.

[0014] Furthermore, the shared feature fusion includes:

[0015] The system consists of two cross-attention fusion modules and a feedforward neural network. First, pairwise fusion is performed: based on task relevance, facial features and pose features are used as query vectors, and interaction behavior features are used as key and value vectors. Preliminary fused features are obtained through cross-attention and weighted summation. After obtaining the two sets of fused representations, "face-interaction" and "pose-interaction," they are concatenated to integrate different semantic information. Subsequently, the concatenated features are fed into the feedforward network (FFN) for transformation, and another path serves as a residual connection. Finally, the two are element-wise added to obtain the enhanced final shared feature fusion result, calculated as follows:

[0016]

[0017]

[0018] In the formula: This indicates the cross-attention feature fusion module. , , Represents interactive behavior, facial features, and posture characteristics; This indicates the result of shared feature fusion.

[0019] Furthermore, a reweighted multi-task loss function is used to dynamically balance the gradients of sub-tasks. This loss function unifies the gradient scale of each task through gradient normalization and adjusts the loss weights based on the correlation coefficient calculated according to the cosine similarity of gradients between tasks.

[0020] First, gradient normalization is used to ensure that the gradient norms of different tasks are at a similar scale:

[0021]

[0022] in, Indicates the target gradient scale; Represents the gradient norm for each task;

[0023] The loss weights are dynamically adjusted using the gradient cosine similarity between tasks. If the gradient directions of two tasks are similar, it indicates that their optimization objectives are consistent, and they are optimized together. Conversely, if the gradient directions are opposite, the loss weights of conflicting tasks are reduced. The loss function of the multi-task learning model is as follows:

[0024]

[0025]

[0026]

[0027] in, Represents the correlation coefficient of subtask i; Let represent the gradient cosine similarity between subtasks i and j; Represents the multi-task learning weights of subtask i; It is a small positive number used to prevent division by zero errors; Let be the weight coefficient for the i-th task; This represents the loss function value for the i-th task.

[0028] Furthermore, the construction of a heterogeneous graph containing expression nodes, pose nodes, and object nodes from the obtained enhanced shared features includes:

[0029] A heterogeneous graph G={V,E,T} of learning states is constructed; this graph contains three types of nodes: expression nodes, pose nodes, and object nodes; the features of all nodes are vectors extracted by the proposed multi-task recognition method; the connection methods between nodes vary according to the different relationships: the edge types between object nodes are divided into three forms based on the human-object interaction recognition results: no interaction, hand interaction, or eye-line interaction; while expression and pose nodes are fully connected to all object nodes.

[0030] Furthermore, based on the constructed heterogeneous graph, hierarchical feature extraction is used for feature encoding to output the reasoning intent, including:

[0031] Hierarchical feature encoding includes hierarchical feature extraction and hierarchical feature gating fusion. Hierarchical feature extraction is used to capture information at different levels, while hierarchical feature gating fusion adaptively controls the fusion method of features at different levels.

[0032] Hierarchical feature extraction extracts hierarchical features from the raw data, including first-level label features and second-level label features. This module consists of three heterogeneous graph convolutional layers. The heterogeneous graph convolutional layers use different graph neural network transformations for different meta-paths. Specifically, in a single-layer network, three independent GCNConv networks are used for feature extraction for three interaction edge types between object nodes: no interaction, hand interaction, and eye-eye interaction. For the relationship between expression or pose nodes and object nodes, the GATConv network structure is used for feature updating and information propagation. All messages generated by the meta-paths are aggregated by addition as the final output of the network layer.

[0033] The hierarchical feature gating fusion module is responsible for fusing intent features from different levels. Through a gating mechanism, it adaptively regulates and dynamically adjusts the contribution weights of intent features from different levels.

[0034] Furthermore, the contrastive learning loss function is as follows:

[0035] The improved reweighted contrast loss function first calculates the contrast loss between the anchor sample and the positive sample pair of the same class, and then obtains the relative loss value through normalization. Specifically, for a sample set within a batch, the sample's own features are used as anchors to form positive sample pairs with samples of the same class. The basic contrast loss uses a softmax function adjusted by the temperature coefficient τ to calculate the similarity between samples.

[0036]

[0037] in, Temperature is a parameter that controls the discrimination of the sample distribution. Indicates sample and The contrast loss between them;

[0038] A weight adjustment mechanism based on the Focal Loss concept is introduced: first, the loss of positive sample pairs is normalized to the maximum value, and then γ exponential weights are applied.

[0039]

[0040]

[0041] in, This represents the weighting adjustment factor; ultimately, the reweighted contrastive loss takes the form of:

[0042]

[0043] The final overall contrast loss is obtained by weighted summation of the contrast losses of the two levels of labels;

[0044]

[0045] in, and It is a hyperparameter used to balance the loss contributions of primary and secondary labels. and Let represent the contrastive loss functions for the first-level labels and the second-level labels, respectively.

[0046] Secondly, the present invention provides a learner intent reasoning system for online learning scenarios, comprising:

[0047] The data acquisition module is used to acquire images of online learners and simultaneously perform head pose recognition, torso sitting posture recognition, and interactive behavior recognition through a multi-task joint training framework to obtain enhanced shared features.

[0048] The heterogeneous graph construction module is used to construct a heterogeneous graph containing expression nodes, pose nodes, and object nodes from the obtained enhanced shared features;

[0049] The inference output module is used to encode features based on the constructed heterogeneous graph using hierarchical feature extraction and output the inference intent.

[0050] Thirdly, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the learner intent reasoning method for online learning scenarios.

[0051] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the learner intent reasoning method for online learning scenarios.

[0052] Compared with the prior art, the present invention has the following technical effects:

[0053] The technical solution provided by this invention effectively solves several key problems existing in current online learning intent reasoning. Addressing the performance limitations of single-task learning due to its neglect of the intrinsic correlation between facial expressions, postures, and interactive behaviors, this invention employs a multi-task joint training framework to simultaneously perform head posture, torso posture, and interactive behavior recognition. Pre-trained InceptionResNet, Openpose, and Detr expert networks are used to extract facial expression, posture, and interactive behavior features respectively. Furthermore, a cross-attention fusion module and a feedforward neural network are used to achieve pairwise feature fusion and enhance the construction of shared features. This not only captures the correlation between different tasks but also dynamically balances sub-task gradients through gradient normalization and a reweighted multi-task loss function based on gradient cosine similarity, avoiding optimization difficulties caused by task conflicts and thus improving the model's ability to recognize complex learning states.

[0054] To address the challenges posed by the hierarchical nature of online learning intentions and the imbalanced distribution of long-tailed datasets, the technical solution constructs a heterogeneous graph that includes facial expressions, poses, and object nodes. It then uses three layers of heterogeneous graph convolutional layers to extract features from different meta-paths using independent GCNConv or GATConv networks. This effectively captures fine-grained information such as no interaction between objects, hand interactions, and eye-to-eye interactions. Furthermore, by integrating facial expression and pose node information through a fully connected approach, it achieves a comprehensive expression of multi-level semantic features.

[0055] Furthermore, the hierarchical feature gating fusion module adaptively adjusts the contribution weights of features at different levels, addressing the issue of traditional methods performing poorly on long-tailed and small-sample data, thus further improving the accuracy of intent inference. The improved reweighted contrastive loss function enhances the model's ability to distinguish between similar samples by introducing a temperature coefficient and a Focal Loss weight adjustment mechanism. Simultaneously, the weighted summation of the two-level label contrastive loss balances the optimization objectives of features at different levels, thereby achieving more refined semantic expression and higher intent inference accuracy in complex online learning scenarios. Attached Figure Description

[0056] Figure 1 Multi-task joint training framework.

[0057] Figure 2 Schematic diagram of multi-scale feature fusion.

[0058] Figure 3 A schematic diagram of the construction of a heterogeneous learning state graph.

[0059] Figure 4 Schematic diagram of the hierarchical feature encoder structure.

[0060] Figure 5 Schematic diagram of the hierarchical feature extraction module.

[0061] Figure 6 Schematic diagram of hierarchical feature gating fusion module.

[0062] Figure 7 OpenPose model structure diagram.

[0063] Figure 8 Structure diagram of the pre-trained Detr model.

[0064] Figure 9 This is a flowchart of the present invention. Detailed Implementation

[0065] The present invention will be further described below with reference to the accompanying drawings:

[0066] Example 1, please refer to Figure 9 This invention provides a learner intent reasoning method for online learning scenarios, comprising:

[0067] Images of online learners are acquired, and head pose recognition, torso sitting posture recognition, and interactive behavior recognition are performed simultaneously through a multi-task joint training framework to obtain enhanced shared features.

[0068] The obtained enhanced shared features are used to construct a heterogeneous graph containing expression nodes, pose nodes, and object nodes;

[0069] Based on the constructed heterogeneous graph, hierarchical feature extraction is used for feature encoding to output the reasoning intent.

[0070] This invention identifies learners' learning intentions based on online learner images. Online learning images present challenges such as isolated processing of multimodal features, insufficient perception of key interaction points, and imbalanced intent data categories. To address these challenges, a multi-task joint training framework is used to simultaneously process three sub-tasks: head pose recognition, torso posture recognition, and interactive behavior recognition. Based on this framework, a heterogeneous learning state graph is constructed, and further supervised contrastive learning is used to infer learning intentions.

[0071] Example 2: This invention provides a learner intent reasoning method for online learning scenarios, comprising:

[0072] 1) Multi-task recognition of learner learning status

[0073] In online learning scenarios, learner image data primarily depicts the learner's upper body. This invention employs a multi-task joint training framework to simultaneously process three sub-tasks: head pose recognition, torso posture recognition, and interactive behavior recognition. An attention-based shared feature fusion module is used to achieve cross-enhancement of facial, pose, and interactive features, and a reweighted multi-task loss function is used to balance the gradient updates of the sub-tasks.

[0074] 2) Construction of learner state heterogeneity graph

[0075] This invention, based on the features output by the multi-task recognition module, represents the learning state as a heterogeneous graph containing learner expression nodes, pose nodes, and object nodes. The connections between nodes are divided into three categories: object-to-object edges, expression-to-object edges, and pose-to-object edges. Among them, object-to-object edges are further divided into three types based on the interaction recognition results: no interaction, hand interaction, and eye-to-eye interaction. Expression / pose-to-object edges adopt a fully connected approach.

[0076] 3) Learner Intention Inference

[0077] Based on the construction of a heterogeneous graph of learner state, this invention uses a heterogeneous graph network and a gating feature fusion mechanism to dynamically mix the original features of learning intent with low-level features. Finally, it optimizes the feature space through comparative learning, thereby further predicting the intent inference result.

[0078] The multi-task joint training framework in step 1) is as follows: Figure 1 As shown.

[0079] The multi-task joint training framework mainly consists of five stages: image input, feature extraction, feature fusion, task classification, and result output. In the feature extraction stage, facial expression, pose, and interaction behavior subtasks utilize pre-trained InceptionResNet, OpenPose, and Detr expert networks to extract relevant task features, respectively. In the feature fusion stage, a shared feature fusion module is used to perform shared feature fusion on relevant features. In the task classification stage, different classification heads are used to learn and classify the relevant task features fused with shared features, and the multi-task loss is calculated. Finally, the recognition results for each task are output.

[0080] The feature classification stage classifier consists of a fully connected layer and a Softmax function, effectively distinguishing between different categories of facial expressions and poses. In the interactive behavior recognition task, interactive behavior features incorporating shared features are used as input and further fed into the classification module for recognition. This module, composed of the explicit query module and Transformer decoder mentioned earlier, possesses strong context modeling and semantic parsing capabilities, enabling accurate recognition of multiple interactive behavior categories.

[0081] The shared feature fusion module in step 1) is shown below:

[0082] The shared feature fusion stage mainly consists of two cross-attention fusion modules and a feedforward neural network. This stage first performs pairwise fusion: based on task relevance, facial features and pose features are used as query vectors, and interaction behavior features are used as key and value vectors. Preliminary fused features are obtained through cross-attention and weighted summation. To ensure stable and efficient training, residual connections and normalization operations are also introduced. After obtaining the two sets of fused representations, "face-interaction" and "pose-interaction," they are concatenated to integrate different semantic information. Subsequently, the concatenated features are fed into the feedforward network (FFN) for transformation, and another path serves as a residual connection. Finally, the two are element-wise added to obtain the enhanced final shared feature fusion result, calculated as follows:

[0083]

[0084]

[0085] In the formula: This indicates the cross-attention feature fusion module. , , It represents interactive behavior, facial features, and posture characteristics. This indicates the result of shared feature fusion.

[0086] The explicit query module in step 1) is shown below:

[0087] The explicit query module constructs query vectors by fusing content priors and multi-scale spatial prior information. This module is implemented through two parallel branches. The first is a spatial feature extraction branch, which uses candidate boxes and classification features output by the Detr model to calculate and stack spatial relationship features (including relative center position, relative size, intersection-over-union ratio, etc.) for effective person-object pairing. Then, a multilayer perceptron (MLP) is used for nonlinear transformation to extract interaction-related spatial information. The second is a semantic feature extraction branch, which obtains high-dimensional object features output by the Detr encoder and inputs them into the Transformer Encoder to extract high-level semantics. The key to this branch is the introduction of a multi-scale feature fusion structure, which fuses semantic features at different scales by constructing a feature pyramid, thereby enhancing the model's modeling capabilities.

[0088] in For the feature representation output by the Transformer Encoder, Input features for the next level of scale features This represents the features during the convolution process of an object image. This represents the number of hidden layers. The calculation formula is shown below.

[0089]

[0090] Finally, after processing by the spatial feature extraction branch and the semantic feature extraction branch, spatial features and semantic features are obtained respectively. Next, the two types of features are initially fused through feature concatenation, and the concatenated result is input into the MLP network for feature transformation, ultimately obtaining the construction result of the explicit query.

[0091] The reweighted multi-task loss function in step 1) is shown below:

[0092] To address the issues of large differences in the scale of subtasks and conflicting optimization directions in multi-task learning, this invention utilizes gradient information to dynamically adjust the loss weights of each task, thereby balancing the progress of multi-task learning and reducing the impact of gradient conflicts. This method includes two mechanisms: gradient normalization and correlation weighting. First, gradient normalization ensures that the gradient norms of different tasks are at similar scales, preventing the gradient of one task from dominating the optimization process.

[0093]

[0094] in, Indicates the target gradient scale; This represents the gradient norm for each task.

[0095] To address the gradient conflict and convergence interference issues caused by differing optimization objectives in multi-task learning, this paper employs the cosine similarity of gradients between tasks to dynamically adjust the loss weights. The basic principle is: if the gradient directions of two tasks are similar, it indicates that their optimization objectives are consistent and they can be optimized together; conversely, if the gradient directions are opposite, it indicates a conflict between the tasks, and the loss weight of the conflicting task should be reduced to mitigate its negative impact. This strategy is implemented through a correlation coefficient as shown below. Based on this parameter design, the loss function of the multi-task learning model designed in this invention is as follows:

[0096]

[0097]

[0098]

[0099] in, Represents the correlation coefficient of subtask i; Let represent the gradient cosine similarity between subtasks i and j. Represents the multi-task learning weights of subtask i; It is a small positive number used to prevent division by zero errors. Let be the weight coefficient for the i-th task; This represents the loss function value for the i-th task.

[0100] The heterogeneous graph of the learning states in step 2) is constructed as follows:

[0101] To fully model the complex relationships between learners' facial expressions, postures, and interactions with objects, this study constructs a heterogeneous learning state graph G={V,E,T}. This graph contains three types of nodes: facial expression nodes, posture nodes, and object nodes (including people and interactive objects). The features of all nodes are 512-dimensional vectors extracted using the multi-task recognition method proposed in step 1). The connection methods between nodes vary depending on the relationship: the edge types between object nodes are categorized into three forms based on the human-object interaction recognition results: no interaction, hand interaction, or eye-to-eye interaction; while facial expression and posture nodes are fully connected to all object nodes to allow the model to learn the complex implicit associations during training. The final heterogeneous graph is shown below. Figure 3 As shown, this provides a comprehensive state representation for subsequent intention reasoning.

[0102] In step 3), heterogeneous graph networks such as Figure 4 As shown.

[0103] In step 3), the goal of the heterogeneous graph network is to extract hierarchical feature representations from the input data and to fuse information from different levels through a gating mechanism to improve the learning effect of multi-level features. This invention designs a hierarchical feature encoder with an overall structure comprising two key modules: a hierarchical feature extraction module and a hierarchical feature gating fusion module. The former captures information from different levels, while the latter adaptively controls the fusion method of features from different levels to alleviate information conflicts caused by direct concatenation.

[0104] Step 3) involves a hierarchical feature extraction module responsible for extracting hierarchical features from the raw data, including first-level and second-level label features. This module employs a three-layer heterogeneous graph convolutional layer. To handle heterogeneous graphs containing different types of nodes and connections, such as expressions, poses, and objects, the heterogeneous graph convolutional layer designed in this paper uses different graph neural network transformations for different meta-paths. Specifically, in a single-layer network, three independent GCNConv networks are used for feature extraction for the three types of interaction edges between object nodes (no interaction, hand interaction, and eye contact); while for the relationship between expression or pose nodes and object nodes, a GATConv network structure is used for feature updating and information propagation. Finally, the messages generated by all meta-paths are aggregated by addition as the final output of this network layer. The module structure is as follows: Figure 5 As shown.

[0105] Step 3) involves a hierarchical feature gating fusion module responsible for fusing intent features from different levels. Its core function is to adaptively control and dynamically adjust the contribution weights of intent features from different levels through a gating mechanism. This design enables the model to selectively select and fuse information based on different heterogeneous graph nodes or samples. Its specific structure is as follows: Figure 6 As shown.

[0106] The contrastive learning loss function in step 3) is shown below:

[0107] To address the issue of traditional supervised contrastive loss neglecting differences in sample difficulty, this invention proposes an improved reweighted contrastive loss function. This method first calculates the contrastive loss between the anchor sample and a pair of positive samples of the same class, and then obtains the relative loss value through normalization. Specifically, for a sample set within a batch, the sample's own features are used as anchors, forming positive sample pairs with samples of the same class. The basic contrastive loss uses a softmax function adjusted by a temperature coefficient τ to calculate the similarity between samples.

[0108]

[0109] in, Temperature is a parameter that controls the discrimination of the sample distribution. Indicates sample and The difference in loss between them.

[0110] To enhance the model's attention to difficult samples, a weight adjustment mechanism based on the Focal Loss concept is introduced: first, the loss of positive sample pairs is normalized to the maximum value, and then γ exponential weights are applied.

[0111]

[0112]

[0113] in, This represents the weight adjustment factor. Finally, the reweighted contrastive loss takes the form:

[0114]

[0115] The final overall contrast loss is obtained by weighted summation of the contrast losses of the two levels of labels.

[0116]

[0117] in, and It is a hyperparameter used to balance the loss contributions of primary and secondary labels. and Let represent the contrastive loss functions for the first-level labels and the second-level labels, respectively.

[0118] Example 3: This invention provides a learner intent reasoning method for online learning scenarios, including:

[0119] 1) Learning State Feature Extraction Network Structure

[0120] Facial features were extracted using a pre-trained InceptionResNetV1 model. This model was trained on the large-scale face dataset VGGFace2, and its network structure consists of three Inception-ResNet blocks, as detailed below:

[0121] Table 1. Facial Feature Extraction Network Structure

[0122]

[0123] Pose features were extracted using the OpenPose model, and the model structure is as follows: Figure 7 As shown:

[0124] OpenPose uses the first ten layers of VGG-19 as its backbone network. The 64-dimensional feature map output of the backbone network serves as the input for subsequent keypoint detection and keypoint association networks. This results in an N×4-dimensional keypoint candidate list and an N×20-dimensional skeleton structure index. Based on the skeleton structure index, the keypoint candidate list is then organized into an 18*3-dimensional pose feature.

[0125] Interaction behavior features are extracted using a pre-trained Detr model, the model structure of which is as follows: Figure 8 As shown:

[0126] The Detr module first extracts features from the input image using a CNN, converting the raw image into a feature map to capture visual information. Then, the feature map is fed into the Transformer architecture. The final output is a 512-dimensional feature map of the interactive behavior.

[0127] 2) Explanation of the effects

[0128] (1) Explanation of the comparative model

[0129] This invention compares the interactive behavior recognition performance with the QPIC algorithm, CDN, and PViC algorithm, and the pose recognition performance with the traditional ResNet convolutional network and OpenPose method. It also compares the performance of expression recognition with the classic ResNet and DAN networks as the baseline methods.

[0130] (2) Comparison of interactive behavior recognition performance

[0131] The experimental results comparing the method of this invention with the QPIC algorithm, CDN, and PViC algorithm on an online learning dataset are shown in Table 2.

[0132] Table 2 Comparison of Interactive Behavior Recognition Results

[0133]

[0134] Experiments have shown that the interactive behavior recognition effect of the present invention is better, with an average precision (mAP) of 56.5 on the online learning dataset.

[0135] (3) Comparison of pose recognition accuracy

[0136] The experimental results of this invention for posture recognition are shown in Table 3. Table 3 shows the recognition accuracy of different methods for learner postures.

[0137] Table 3 Comparison of Pose Recognition Results

[0138]

[0139] (4) Comparison of facial expression recognition accuracy

[0140] The experimental results of this invention for facial expression recognition are shown in Table 4. Table 4 shows the recognition accuracy of different methods for learner facial expressions.

[0141] Table 4 Comparison of Facial Expression Recognition Results

[0142]

[0143] (5) Comparison of learning intention reasoning effects

[0144] This invention was compared with the MESA, SMOTEWB, TLC, AnchorAL, and ECGN algorithms to demonstrate its effectiveness in learning intent reasoning. The experimental results are shown in Table 5. Analysis of the experimental results on two datasets shows that the model achieved optimal results across all metrics. The two-stage intent reasoning method performs better for overall intent reasoning in imbalanced datasets, achieving precisions of 76.2% and 81.5% on the OGB and OLID datasets, respectively; and F1 scores of 48.2% and 71.6%, respectively, demonstrating the advantages of this invention for online intent reasoning learning.

[0145] Table 5 Comparison of Experimental Results

[0146]

[0147] In another embodiment of the present invention, a learner intent reasoning system for online learning scenarios is provided, which can be used to implement the above-mentioned learner intent reasoning method for online learning scenarios. Specifically, the system includes:

[0148] The data acquisition module is used to acquire images of online learners and simultaneously perform head pose recognition, torso sitting posture recognition, and interactive behavior recognition through a multi-task joint training framework to obtain enhanced shared features.

[0149] The heterogeneous graph construction module is used to construct a heterogeneous graph containing expression nodes, pose nodes, and object nodes from the obtained enhanced shared features;

[0150] The inference output module is used to encode features based on the constructed heterogeneous graph using hierarchical feature extraction and output the inference intent.

[0151] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0152] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions from the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a learner intent reasoning method for online learning scenarios.

[0153] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the learner intent reasoning method for online learning scenarios described in the above embodiments.

[0154] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0156] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A learner intention reasoning method for an online learning scenario, characterized in that, include: Images of online learners are acquired, and head pose recognition, torso sitting posture recognition, and interactive behavior recognition are performed simultaneously through a multi-task joint training framework to obtain enhanced shared features. The obtained enhanced shared features are used to construct a heterogeneous graph containing expression nodes, pose nodes, and object nodes; Based on the constructed heterogeneous graph, hierarchical feature extraction is used for feature encoding to output the reasoning intent; The acquisition of online learner images, and the simultaneous execution of head pose recognition, torso sitting posture recognition, and interaction behavior recognition through a multi-task joint training framework, includes: The multi-task joint training framework consists of five stages: image input, feature extraction, feature fusion, task classification, and result output. In the feature extraction stage, the expression, pose, and interaction behavior sub-tasks are extracted using pre-trained InceptionResNet, OpenPose, and Detr expert networks, respectively. In the feature fusion stage, shared feature fusion is performed on the relevant features. In the task classification stage, different classification heads are used to learn the classification of the relevant task features fused with shared features, and the multi-task loss is calculated. Finally, the recognition results of each task are output. The classifier in the feature classification stage consists of a fully connected layer and a Softmax function, which effectively distinguishes different categories of facial expressions and poses. The shared feature fusion includes: The system consists of two cross-attention fusion modules and a feedforward neural network. First, pairwise fusion is performed: based on task relevance, facial features and pose features are used as query vectors, and interaction behavior features are used as key and value vectors. Preliminary fused features are obtained through cross-attention and weighted summation. After obtaining the two sets of fused representations, "face-interaction" and "pose-interaction," they are concatenated to integrate different semantic information. Subsequently, the concatenated features are fed into the feedforward network (FFN) for transformation, and another path serves as a residual connection. Finally, the two are element-wise added to obtain the enhanced final shared feature fusion result, calculated as follows: In the formula: represents a cross-attention feature fusion module, 、 、 represents an interaction behavior, face, and posture feature; represents a shared feature fusion result; Based on a reweighted multi-task loss function that dynamically balances the gradients of subtasks, this loss function unifies the gradient scale of each task through gradient normalization and adjusts the loss weights based on the correlation coefficient calculated according to the cosine similarity of gradients between tasks. First, gradient normalization is used to ensure that the gradient norms of different tasks are at a similar scale: wherein, denotes the target gradient scale; denotes the gradient norm of each task; The loss weights are dynamically adjusted using the gradient cosine similarity between tasks. If the gradient directions of two tasks are similar, it indicates that their optimization objectives are consistent, and they are optimized together. Conversely, if the gradient directions are opposite, the loss weights of conflicting tasks are reduced. The loss function of the multi-task learning model is as follows: wherein, represents the correlation coefficient of subtask i; represents the gradient cosine similarity of subtasks i and j; represents the multi-task learning weight of subtask i; is a small positive number, used to prevent division by zero error; is the weight coefficient of the i-th task; represents the loss function value of the i-th task. 2.The learner intention reasoning method for online learning scenarios according to claim 1, characterized in that, The process of constructing a heterogeneous graph containing expression nodes, pose nodes, and object nodes from the obtained enhanced shared features includes: A heterogeneous graph G={V,E,T} of learning states is constructed; this graph contains three types of nodes: expression nodes, pose nodes, and object nodes; the features of all nodes are vectors extracted by the proposed multi-task recognition method; the connection methods between nodes vary according to the different relationships: the edge types between object nodes are divided into three forms based on the human-object interaction recognition results: no interaction, hand interaction, or eye-line interaction; while expression and pose nodes are fully connected to all object nodes. 3.The learner intention reasoning method for online learning scenarios according to claim 1, characterized in that, The constructed heterogeneous graph uses hierarchical feature extraction for feature encoding and outputs the reasoning intent, including: Hierarchical feature encoding includes hierarchical feature extraction and hierarchical feature gating fusion. Hierarchical feature extraction is used to capture information at different levels, while hierarchical feature gating fusion adaptively controls the fusion method of features at different levels. Hierarchical feature extraction extracts hierarchical features from the original data, including first-level and second-level label features. It employs a three-layer heterogeneous graph convolutional layer. The heterogeneous graph convolutional layer uses different graph neural network transformations for different meta-paths. Specifically, in a single-layer network, three independent GCNConv networks are used for feature extraction for three interaction edge types between object nodes: no interaction, hand interaction, and eye-tracking interaction. For the relationship between expression or pose nodes and object nodes, a GATConv network structure is used for feature updating and information propagation. Messages generated by all meta-paths are aggregated by addition as the final output of the network layer. The hierarchical feature gating fusion module is responsible for fusing intent features from different levels. Through a gating mechanism, it adaptively regulates and dynamically adjusts the contribution weights of intent features from different levels.

4. The learner intention reasoning method for online learning scenarios according to claim 3, characterized in that, The contrastive learning loss function is as follows: The improved reweighted contrast loss function first calculates the contrast loss between the anchor sample and the positive sample pair of the same class, and then obtains the relative loss value through normalization. Specifically, for a sample set within a batch, the sample's own features are used as anchors to form positive sample pairs with samples of the same class. The basic contrast loss uses a softmax function adjusted by the temperature coefficient τ to calculate the similarity between samples. wherein, is a temperature parameter, controlling the degree of separation of the sample distribution; denotes the contrast loss between the sample and the ground truth. A weight adjustment mechanism based on the Focal Loss concept is introduced: first, the loss of positive sample pairs is normalized to the maximum value, and then γ exponential weights are applied. wherein, denotes the weight adjustment factor; finally, the re-weighted contrastive loss takes the form of: The final overall contrast loss is obtained by weighted summation of the contrast losses of the two levels of labels; wherein, and are hyperparameters for balancing the contribution of the primary and secondary label losses; and denote the contrastive loss functions for the primary and secondary labels, respectively.

5. A learner intent reasoning system for online learning scenarios, characterized in that, include: The data acquisition module is used to acquire images of online learners and simultaneously perform head pose recognition, torso sitting posture recognition, and interactive behavior recognition through a multi-task joint training framework to obtain enhanced shared features. The heterogeneous graph construction module is used to construct a heterogeneous graph containing expression nodes, pose nodes, and object nodes from the obtained enhanced shared features; The inference output module is used to encode features based on the constructed heterogeneous graph using hierarchical feature extraction and output the inference intent. The acquisition of online learner images, and the simultaneous execution of head pose recognition, torso sitting posture recognition, and interaction behavior recognition through a multi-task joint training framework, includes: The multi-task joint training framework consists of five stages: image input, feature extraction, feature fusion, task classification, and result output. In the feature extraction stage, the expression, pose, and interaction behavior sub-tasks are extracted using pre-trained InceptionResNet, OpenPose, and Detr expert networks, respectively. In the feature fusion stage, shared feature fusion is performed on the relevant features. In the task classification stage, different classification heads are used to learn the classification of the relevant task features fused with shared features, and the multi-task loss is calculated. Finally, the recognition results of each task are output. The classifier in the feature classification stage consists of a fully connected layer and a Softmax function, which effectively distinguishes different categories of facial expressions and poses. The shared feature fusion includes: The system consists of two cross-attention fusion modules and a feedforward neural network. First, pairwise fusion is performed: based on task relevance, facial features and pose features are used as query vectors, and interaction behavior features are used as key and value vectors. Preliminary fused features are obtained through cross-attention and weighted summation. After obtaining the two sets of fused representations, "face-interaction" and "pose-interaction," they are concatenated to integrate different semantic information. Subsequently, the concatenated features are fed into the feedforward network (FFN) for transformation, and another path serves as a residual connection. Finally, the two are element-wise added to obtain the enhanced final shared feature fusion result, calculated as follows: In the formula: This indicates the cross-attention feature fusion module. , , Represents interactive behavior, facial features, and posture characteristics; This indicates the result of shared feature fusion; Based on a reweighted multi-task loss function that dynamically balances the gradients of subtasks, this loss function unifies the gradient scale of each task through gradient normalization and adjusts the loss weights based on the correlation coefficient calculated according to the cosine similarity of gradients between tasks. First, gradient normalization is used to ensure that the gradient norms of different tasks are at a similar scale: in, Indicates the target gradient scale; Represents the gradient norm for each task; The loss weights are dynamically adjusted using the gradient cosine similarity between tasks. If the gradient directions of two tasks are similar, it indicates that their optimization objectives are consistent, and they are optimized together. Conversely, if the gradient directions are opposite, the loss weights of conflicting tasks are reduced. The loss function of the multi-task learning model is as follows: in, Represents the correlation coefficient of subtask i; Let represent the gradient cosine similarity between subtasks i and j; Represents the multi-task learning weights of subtask i; It is a small positive number used to prevent division by zero errors; Let be the weight coefficient for the i-th task; This represents the loss function value for the i-th task.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the learner intent reasoning method for online learning scenarios as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the learner intent reasoning method for online learning scenarios as described in any one of claims 1 to 4.

Images