Visual-textual contrastive embedding and graph relational context fusion for behavior recognition

Abstract

This invention relates to a visual-text contrastive embedding and graph relational context fusion method for behavior recognition. Through contrastive representation learning, synchronous classroom videos and contextual text information are aligned to a shared embedding space. Graph-based relational modeling is introduced to capture dependencies between behavior categories, enhancing multimodal representation. Relation-driven description enhancement and contextual conditional cues generated from IoT metadata are used to enrich text semantics. A contrastive visual-text embedding model (teacher CVTE model) for IoT-enabled student behavior recognition is constructed. A teacher-student knowledge distillation strategy is introduced to transfer the multimodal knowledge from the teacher CVTE model (high-capacity teacher model) in step four to a lightweight student model (student CVTE model), constructing an end-to-end CVTE framework. The method of this invention can achieve low-latency response, high recognition accuracy, and scalable deployment in smart classroom scenarios.

Description

Technical Field

[0001] This invention relates to the field of behavior recognition, and more particularly to a method for visual text contrast embedding and graph relational context fusion in behavior recognition. Background Technology

[0002] Classroom student behavior recognition is crucial for evaluating teaching effectiveness, understanding student learning status, and supporting dynamic, interactive learning environments. With the rapid development of smart classrooms and the widespread application of the Internet of Things (IoT) technology, cameras, microphones, wearable devices, environmental sensors, and classroom management systems can be integrated into a connected educational infrastructure. This IoT-enabled environment continuously generates behavioral and contextual data streams, providing new possibilities for fine-grained classroom analytics. In these scenarios, accurate and efficient student behavior recognition can support teaching evaluation, personalized instruction, engagement monitoring, and timely intervention.

[0003] However, classroom behavior recognition remains a challenging task. Traditional methods typically rely on human observation, static visual data, or unimodal video streams. While these methods have driven the development of automated classroom analytics, they fall short in capturing the temporal complexity, semantic context, and multimodal interactions of real-world classroom activities. Classroom behaviors such as group discussions, collaborative learning, sustained attention, and brief distractions are often difficult to distinguish based on visual appearance alone. They are typically manifested as subtle changes in posture, short-term movement patterns, gaze direction, object interactions, and contextual cues. Therefore, effective classroom behavior recognition requires not only spatial and temporal modeling but also a semantic understanding of the learning context.

[0004] Recent research has attempted to incorporate multimodal learning into student behavior analysis to address the aforementioned limitations. By combining visual data with complementary modalities such as audio, text, or sensor-generated contextual information, multimodal models can form a more comprehensive representation of classroom interactions. However, existing multimodal methods still face several limitations. Audio signals are susceptible to overlapping speech, background noise, and microphone position; pure visual models are often constrained by occlusion, appearance-similar behaviors, and long-term temporal dependencies. Furthermore, many deep learning models are designed for environments with abundant GPU resources, making them difficult to deploy on resource-constrained IoT edge devices. This results in a real gap between recognition accuracy and real-time inference efficiency.

[0005] Visual-language learning offers a promising direction for addressing these issues. Compared to audio signals, textual information derived from behavioral labels, classroom context, teacher instructions, student responses, digital logs, and IoT metadata provides more stable and semantically rich supervisory signals. Contrastive learning further enables visual and textual representations to align within a shared embedding space, thereby improving intra-class compactness and inter-class separability. However, existing visual-text methods are not yet adequately suited for IoT-enabled classroom behavior recognition tasks; such tasks require models to simultaneously handle contextual semantics, behavioral relationships, limited supervision, and lightweight deployment. Summary of the Invention

[0006] The purpose of this invention is to provide a visual text contrast embedding and graph relation context fusion method for behavior recognition, which can achieve low latency response, high recognition accuracy and scalable deployment in smart classroom scenarios.

[0007] To achieve the above objectives, the technical solution adopted by this invention is: a method for behavior recognition based on visual text contrast embedding and graph relation context fusion, comprising the following steps: Step 1: Contrastive visual-text embedding. Through contrastive representation learning, synchronized classroom videos and contextual text information are aligned into a shared embedding space. Step 2: Graph-based relational modeling. Building on Step 1, graph-based relational modeling is introduced to capture the dependencies between behavior categories and enhance multimodal representation. Step 3: Relationship-driven description enhancement. Building on step 2, relationship-driven description enhancement and contextual condition cues generated from IoT metadata are used to enrich the semantics of the text. Step 4: Based on Steps 1 to 3, construct a contrastive visual-text embedding model (teacher CVTE model) for IoT-enabled student behavior recognition. Step 5: Introduce a teacher-student knowledge distillation strategy to transfer the multimodal knowledge from the teacher CVTE model (high-capacity teacher model) in Step 4 to the lightweight student model (student CVTE model), thus constructing an end-to-end CVTE framework.

[0008] Preferably, the CVTE framework consists of three functional layers: a data acquisition layer, an edge layer, and a cloud layer. The data acquisition layer captures synchronous multimodal inputs from IoT sensors, including high-resolution classroom video streams and contextual text logs. The edge layer performs lightweight preprocessing, such as denoising and frame sampling, and performs real-time inference using a compact student CVTE model. The cloud layer receives inference outputs and selected feature summaries from the edge and performs large-scale analysis to improve the interpretability and long-term monitoring capabilities of the system. This layer aggregates temporal behavior patterns, supports time series analysis, and constructs student behavior profiles. It also serves as the training and updating location for the teacher CVTE model.

[0009] Preferably, the CVTE module adopts a CLIP-style dual encoder architecture, and is extended through graph-based relational modeling, relation-driven description enhancement, and contextual condition cues, given a video segment. and its corresponding text description Student visual encoder and student text encoder The input is mapped to a shared feature space, and then passed through a projection head. and Get embedded: The components constitute a lightweight model deployed on resource-constrained IoT edge devices. During the training phase, CVTE also builds a high-capacity teacher network. This network has the same architectural design as the student model, but with greater width and depth, serving as a strong source of supervision for knowledge distillation.

[0010] Preferably, CVTE constructs a scene graph for each video frame or short clip. During training, regions of interest are pre-extracted offline and stored as lightweight annotations. During deployment, edge devices run only the compact CVTE-S backbone network for real-time inference, while YOLOv8 is used only in the offline preprocessing stage to generate a set of regions of interest. These regions are projected as initial node embeddings: Build a relationship graph on top of these entities , among which the side Defined by spatial proximity or semantic prior; optional edge features Used to encode relative position or motion cues, the Graph Attention Network (GAT) then employs standard attention-based message passing, residual connections, and layer normalization. L Information is transmitted and interacted within the layers; after L After the layer, relationship-aware node features Aggregation is performed using the attention-based READOUT operator to obtain a single graph-level embedding: in and As learnable parameters, the final visual embedding incorporates global backbone features. Graph-level features Combining them:

[0011] in Indicating splicing, the same graph modeling module is also instantiated into the teacher network, and relation enhancement visual features are generated for distillation. .

[0012] Preferably, CVTE introduces a relationship-driven description enhancement mechanism to generate auxiliary text descriptions from the interaction graph, with graph-level embedding. Used as a global semantic summary of classroom interactions, subsequently, a lightweight decoder by Generate relation descriptions for conditions : The generated descriptions are encoded using the same text encoder, resulting in... And as an additional text view for the corresponding visual fragment; for those containing N Small batches of fragments, Indicates the first The set of positive samples of text fragments is used to extend the visual-to-text contrast loss into a multi-positive-sample form: in Represents cosine similarity. Temperature parameter; symmetrical text-to-visual loss Similarly defined, the average of the two is denoted as the overall comparison target of the student network. The teacher network used the same contrast target. It can be trained or fine-tuned, and provides semantically rich multimodal representations for distillation.

[0013] Preferably, IoT-enabled classrooms can provide rich contextual metadata, and CVTE uses signals to modulate the text encoder through contextual conditional cues, enabling... This represents the concatenated metadata vector, the context encoder. Will Mapped to One soft cue vector: in For the text embedding dimension, for lexicalized text sequences and its embedding The hint is added as an adaptive prefix to the front of the sequence: This is then fed into the Transformer-based text encoder in the student network to obtain context-aware text embeddings. The same prompting mechanism can also be applied to the teacher network, enabling both models to benefit from IoT-aware semantic conditions.

[0014] Preferably, to regularize contextual hints and avoid overfitting to specific classroom configurations, Frobenius norm penalty is used: in Denotes the Frobenius norm. Control the regularization strength.

[0015] The preferred knowledge distillation scheme from the teacher network to the student network is as follows: for the same input pair Teacher model generates embeddings and corresponding category posterior The student model generates and ， At the prediction level, Kullback-Leibler divergence with temperature scaling is used: in The distillation temperature. Indicates temperature In addition, the feature layer distillation term is used to encourage the student model to simulate the multimodal representation of the teacher model: Finally, the overall training objective for the student network integrates contrastive loss, cue regularization, and distillation: in, and Control the relative contributions of different items.

[0016] The technical effects of this invention are as follows: This application proposes CVTE, a contrastive visual-text embedding framework that aligns classroom videos with contextual text information into a shared semantic space for IoT-enabled student behavior recognition.

[0017] This application enhances multimodal representation learning through graph-based relation modeling, relation-driven description enhancement, and contextual condition cues generated from IoT metadata, enabling the model to better capture classroom-specific behavioral semantics.

[0018] This application introduces a teacher-student knowledge distillation strategy to transfer multimodal knowledge from a high-capacity teacher model to a lightweight student model, achieving a good balance between recognition accuracy and real-time reasoning efficiency. Attached Figure Description

[0019] Figure 1The overall architecture of the CVTE framework for IoT-enabled student behavior recognition.

[0020] Figure 2 This is a feature visualization diagram based on t-SNE.

[0021] Figure 3 This is the confusion matrix diagram for CVTE-T.

[0022] Figure 4 This is a graph showing the relationship between Top-1 accuracy and inference speed. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be described in detail below with reference to the accompanying drawings. The description in this part is only exemplary and explanatory, and should not be used to limit the scope of protection of the present invention in any way.

[0024] like Figure 1 As shown, this application presents the overall framework architecture of a visual text contrast embedding and graph relational context fusion method for behavior recognition.

[0025] The proposed CVTE framework is designed as an end-to-end architecture specifically for IoT-enabled classroom environments, as shown in Figure 1. The framework consists of three functional layers.

[0026] The data acquisition layer (End) captures synchronous multimodal inputs from IoT sensors, including high-resolution classroom video streams and contextual text logs (such as course information and digital activity records). This data is securely transmitted through a communication gateway, which is responsible for basic protocol conversion and secure routing between local devices and the edge platform.

[0027] The edge layer performs lightweight preprocessing, such as denoising and frame sampling, and uses a compact student CVTE model for real-time inference. This student model is obtained by knowledge distillation from a high-capacity teacher network, thus preserving the discriminative power of rich multimodal representations while meeting edge device latency and computational constraints.

[0028] The cloud layer receives inference output and selected feature summaries from the edge and performs large-scale analysis to improve the system's interpretability and long-term monitoring capabilities. This layer aggregates temporal behavior patterns, supports time series analysis, and constructs student behavior profiles. It also serves as the training and updating location for teacher CVTE models.

[0029] By distributing data collection, inference, and analysis across three levels, the CVTE framework enables low-latency response, high recognition accuracy, and scalable deployment in smart classroom scenarios. Its detailed process is illustrated in the algorithm below.

[0030] 1. Input: Dataset Batch size Learning rate

[0031] 2. Hyperparameters:

[0032] 3. Output: Student parameters after training

[0033] 4. Initialize teacher parameters With student parameters

[0034] 5. For each training iteration, = 1 to MaxIters do 6. Sample small batch index sets

[0035] 7. By Build a scene graph And obtain relation-aware visual features

[0036] 8. By Computational graph embedding Generate relation descriptions and encoded as

[0037] 9. From metadata Calculation software tips and obtain context-aware text embeddings.

[0038] 10. Input the teacher network and get

[0039] 11. By Calculate students

[0040] 12. Construct the set of positive samples And calculate the contrast loss according to equation (6).

[0041] 13. Calculate the prompt regularization term according to formula (9).

[0042] 14. Calculate according to equations (10) and (11) and

[0043] 15. Calculate the total loss according to formula (12).

[0044] 16. Update student parameters:

[0045] 17. End for 18. return

[0046] The framework employs the following technical methods: Contrast Visual-Text Embedding The proposed CVTE module adopts a CLIP-style dual encoder architecture and is extended through graph-based relation modeling, relation-driven description enhancement, and contextual condition hints.

[0047] Given video clip and its corresponding text description Student visual encoder and student text encoder The input is mapped to a shared feature space, and then passed through a projection head. and Get embedded: The components constitute a lightweight model deployed on resource-constrained IoT edge devices. During the training phase, CVTE also builds a high-capacity teacher network. This network shares the same architectural design as the student model, but with greater width and depth, serving as a strongly supervised source for knowledge distillation; in the following text, for simplicity, the student branch will be used and superscripts will be omitted. Unless otherwise specified, and Both represent student embeddings, and the teacher network is optimized using the same contrastive objective; its representation is used only for distillation.

[0048] CVTE uses contrastive loss to align matching visual-text pairs while separating mismatched samples, thus forming the basis for multimodal behavioral representations.

[0049] Graph-based relational modeling To explicitly encode structured classroom interactions, CVTE constructs a scene graph for each video frame or short clip. During training, Regions of Interest (RoIs) are pre-extracted offline and stored as lightweight annotations. During deployment, edge devices run only the compact CVTE-S backbone network for real-time inference, while YOLOv8 is used only in the offline preprocessing stage to generate a set of RoIs. (e.g., students, teachers, and objects), these regions are projected as initial node embeddings:

[0050] Build a relationship graph on top of these entities , among which the side Defined by spatial proximity or semantic priors (e.g., student-teacher, student-object). Optional edge features. Used to encode relative position or motion cues. Subsequently, the Graph Attention Network (GAT) utilizes standard attention-based message passing, residual connections, and layer normalization to... L Interactive information is propagated within the layers. (After...) L After the layer, relationship-aware node features Aggregation is performed using the attention-based READOUT operator to obtain a single graph-level embedding:

[0051] in and As learnable parameters, the final visual embedding incorporates global backbone features. Graph-level features Combining them:

[0052] in Indicating splicing, the same graph modeling module is also instantiated into the teacher network, and relation enhancement visual features are generated for distillation. .

[0053] To further utilize the structural information encoded in the scene graph, CVTE introduces a relationship-driven description enhancement mechanism that generates auxiliary text descriptions from the interaction graph, resulting in the graph-level embedding described above. Used as a global semantic summary of classroom interactions, subsequently, a lightweight decoder (e.g., GRU or Transformer decoder) with Generate relation descriptions for conditions

[0054]

[0055] The generated descriptions are encoded using the same text encoder, resulting in... And serve as an additional text view for the corresponding visual fragment.

[0056] For including N Small batches of fragments, Indicates the first A set of positive text samples (including manually annotated text and relation-driven descriptions) for each segment. The visual-to-text contrastive loss is extended to a multi-positive-sample form:

[0057] in Represents cosine similarity. For temperature parameters. Symmetrical text to visual loss. Similarly defined, the average of the two is denoted as the overall comparison target of the student network. The teacher network used the same contrast target. It can be trained or fine-tuned, and provides semantically rich multimodal representations for distillation.

[0058] Contextual cues from IoT metadata enable IoT-enabled classrooms to provide rich contextual metadata, such as course subject, time period, classroom ID, seating arrangement, and environmental signals; CVTE utilizes these signals to modulate the text encoder through contextual cues; This represents the concatenated metadata vector, the context encoder. Will Mapped to K One soft cue vector:

[0059] in For the text embedding dimension, for lexicalized text sequences and its embedding These hints are added as adaptive prefixes to the front of the sequence:

[0060] This is then fed into the Transformer-based text encoder in the student network to obtain context-aware text embeddings. The same prompting mechanism can also be applied to the teacher network, enabling both models to benefit from IoT-aware semantic conditions.

[0061] To regularize contextual hints and avoid overfitting to specific classroom configurations, this application employs Frobenius norm penalty:

[0062] in Denotes the Frobenius norm. Control the regularization strength.

[0063] Knowledge distillation for edge deployment To achieve a good balance between recognition accuracy and computational efficiency on IoT edge devices, CVTE employs a knowledge distillation scheme from the teacher network to the student network, for the same input pair Teacher model generates embeddings and corresponding category posterior The student model generates and ，At the prediction level, Kullback-Leibler divergence with temperature scaling is used: in The distillation temperature. Indicates temperature In addition, the feature layer distillation term is used to encourage the student model to simulate the multimodal representation of the teacher model: Finally, the overall training objective for the student network integrates contrastive loss, cue regularization, and distillation: in , and The design controls the relative contributions of different items. This allows the lightweight student model to inherit rich multimodal behavioral representations learned by the teacher model, while maintaining sufficient efficiency to support real-time reasoning in IoT-enabled classroom environments.

[0064] Related experiments 1. Dataset and Evaluation Metrics The dataset and evaluation protocol used to evaluate CVTE are shown in Table 1. CSBR10 is an IoT-enabled classroom behavior corpus containing 7,517 multi-scene video-audio clips covering ten behavioral categories, used to evaluate fine-grained multimodal recognition of student interaction behaviors in real classroom scenarios. Each clip is accompanied by an associated behavioral label, automatically or manually generated text description, and IoT metadata (such as course information and classroom ID). The dataset is divided into training, validation, and test sets at the video level to avoid time leakage between clips from the same classroom session.

[0065] This application conducted extensive experiments on the CSBR10 dataset and further evaluated it on the UCF-101 and HMDB-51 benchmarks to examine the generalization ability of the proposed method. Top-1 and Top-5 classification accuracy were used as performance metrics for all datasets. For CSBR10, this application also reports a confusion matrix to better reflect the class-by-class action recognition performance under class imbalance conditions. Computational efficiency was quantified by GFLOPs and inference throughput (FPS).

[0066] Table 1. Evaluation datasets and their characteristics

[0067] 2. Implementation details CVTE is configured with separate teacher and student models. The student model (CVTE-S) uses a ViT-B / 32 visual encoder and a 6-layer Transformer text encoder, with embedding dimensions... d=512 This constitutes a lightweight architecture deployed on edge devices. The teacher model (CVTE-T) employs a ViT-L / 14 visual encoder and a 12-layer Transformer text encoder. d=1024 To provide higher-capacity multimodal representations for knowledge distillation, both the teacher and student models use shallow MLP projection heads. and Furthermore, a lightweight GAT with attention-based READOUT is employed for graph-based relation modeling, as described in the Methods section. The relation-driven description generator is constructed using graph embeddings... g G It is a lightweight Transformer decoder implementation for conditions, and uses the same tokenizer and vocabulary as the text encoder.

[0068] For CSBR10, video clips are uniformly sampled into 16 frames, with a spatial resolution of [missing information]. 224×224 Furthermore, data augmentation techniques such as random cropping and horizontal flipping are applied to the visual stream. Text input combines behavioral and relation-driven descriptions, while IoT metadata is processed through an MLP context encoder. Mapping is done using soft hints. The optimizer uses AdamW, coupled with cosine learning rate scheduling and a short warm-up phase. The contrast temperature is fixed at... The prompt regularization weight in equation (9) is set to The distillation coefficient in equation (12) is set as follows: , , These parameters were fine-tuned once on the validation set and remained constant across all reported results. All models were implemented using PyTorch and trained on a GPU5090 server; after the teacher models were trained offline in the cloud, only the distilled student models were deployed to IoT edge devices for real-time inference.

[0069] 3. Overall performance of classroom behavior recognition 3.1 Comparative Analysis on the CSBR10 Dataset Table 2 summarizes the experimental results on CSBR10. In the visual-only model, VideoMAEV2 achieved the highest Top-1 accuracy (83.12%), while other architectures such as SlowFast, X3D, TimeSformer, Swin-B, UniFormerV2, and InternVideo2 exhibited the expected accuracy-complexity tradeoff in the unimodal setting. Building upon these visual baselines, multimodal classroom methods such as HCMT, VTCL-L, CECL, and CSBR further improved recognition performance by utilizing complementary cues beyond the original video; however, these gains came at the cost of significantly increased computation. For example, CSBR achieved a Top-1 accuracy of 85.94% with 992 GFLOPs, but its computational cost was approximately 5.5 times that of VideoMAE V2 (180 GFLOPs). In this context, the proposed CVTE-S only achieved a Top-1 accuracy of 86.57% with 71 GFLOPs. In terms of accuracy, compared to the strongest existing classroom model CSBR, Top-1 accuracy is improved by 0.63 percentage points, while the computational cost is reduced by about 14 times.

[0070] Relationship modeling and IoT conditional cues enable CVTE to capture structured classroom interactions and IoT-aware semantics. The teacher-student distillation scheme further transfers these representations to a lightweight student model, resulting in a more favorable accuracy-efficiency tradeoff. The teacher model CVTE-T further improves performance, achieving 88.23% Top-1 accuracy and 99.68% Top-5 accuracy at 282 GFLOPs, a significant improvement over CSBR, while requiring only about one-third of the computational budget. These improvements mainly come from better multimodal alignment and richer contextualized representations, which are attributed to interaction-aware relationship modeling and metadata conditional cues.

[0071] Table 2. Comparison results on CSBR10

[0072] 3.2 Evaluation on UCF-101 and HMDB-51 benchmarks To further examine CVTE's generalization ability beyond classroom scenarios, this application conducted comparative experiments on the UCF-101 and HMDB-51 action recognition benchmarks. The results are shown in Table 3. On the UCF-101 dataset, CVTE-S and CVTE-T perform comparably to or slightly better than recent state-of-the-art methods such as HCMT, CECL, and CSBR. On the more challenging HMDB-51 dataset, CVTE-S and CVTE-T also achieve competitive or better accuracy compared to these baselines and HACG. Overall, these results demonstrate that combining graph-based relational modeling and contrastive visual-text embeddings with IoT-aware cues can learn multimodal representations that can be reliably transferred from classroom scenarios to general action recognition benchmarks.

[0073] Table 3. Evaluation on UCF-101 and HMDB-51 benchmarks

[0074] 4. Ablation test of CVTE module To verify the contributions of the various innovations in the CVTE framework, this application conducted systematic ablation experiments on CSBR10.

[0075] 4.1 Results of Graph-Based Relationship Modeling The effectiveness of graph-based relationship modeling was evaluated by comparing the same backbone network without scene graph branches with one that enabled scene graph branches. Table 4 shows that enabling graph branches improved Top-1 accuracy by 0.67%. Graph-level embeddings summarize higher-order student-student, student-teacher, and person-object relationships in the visual flow and are integrated into fragment-level representations before classification and contrastive learning. This demonstrates that enriching the visual backbone network with explicit interaction structures can provide complementary relational cues, thereby going beyond simple appearance and motion features to better distinguish between multi-person behaviors and object-mediated behaviors.

[0076] Table 4. Ablation experiments based on graph-based relational modeling

[0077] 4.2 Relationship-Driven Description Enhancement Effect To examine the impact of relation-driven description enhancement on text branching, this application conducted a corresponding analysis. Two variants were compared under the same training protocol: (i) human-only text, i.e., using only human-annotated text descriptions; and (ii) human-text + relation description, i.e., adding graph relation descriptions generated from the scene graph to the human-annotated descriptions. Table 5 shows that after adding relation-driven descriptions, the Top-1 accuracy improved from 82.82% to 83.57%. This indicates that reusing visual scene graphs to generate additional relational text views can provide complementary semantic cues and strengthen the alignment between visual interaction patterns and text representations.

[0078] Table 5. Relationship-driven description-enhanced ablation experiments

[0079] 4.3 Contextual condition hints from IoT metadata To determine which types of IoT metadata are most beneficial for contextual cues in text branches, this application analyzed the contribution of different metadata types. Comparing four variants under the same training protocol, Table 6 shows that adding timetable metadata slightly improved Top-1 accuracy from 83.57% to 83.81%. Further introducing room hierarchy information (classroom ID and seating layout) improved accuracy to 84.32%. Adding environmental signals in addition to timetable and room metadata resulted in an absolute improvement of 1.03% compared to the uncued baseline; this complete metadata variant is denoted as CVTE in subsequent experiments. Overall, these results indicate that all three types of metadata can provide incremental benefits, possibly because they can encode expected activity patterns at different levels of contextual detail, i.e., "what is happening, when it is happening, and where it is happening."

[0080] Table 6. Contextualized Ablation Experiments

[0081] 4.4 Knowledge Distillation Effect of Student Model The role of knowledge distillation on CSBR10 was analyzed by comparing the teacher model (CVTE-T) and several student model variants, with all models using the same training protocol. The student baseline model was trained without distillation, using only the contrastive target and cue regularization. Distilled student models were then equipped with logit layer distillation, feature layer distillation, or a combination of both, with distillation weights selected on the validation set; Table 7 shows the logit layer distillation... and characteristic layer distillation Both can improve the performance of the student model relative to the undistilled baseline; the effect is best when combined, in combination settings Below, the student model achieves a 1.97 percentage point improvement in Top-1 accuracy compared to the baseline without distillation, while maintaining a lightweight architecture; the first line explicitly reports the teacher model CVTE-T, which serves as the upper bound for performance and is not trained using student distillation.

[0082] Table 7. Student Model Knowledge Distillation and Ablation Experiment

[0083] 5. Experimental Analysis 5.1 Feature Visualization To analyze feature separability, this application applied t-SNE to fragment-level embeddings of four variants: backbone baseline, CVTE (containing complete components but without distillation), CVTE-S (distilled student model), and CVTE-T. As shown in Figure 2a, the backbone features exhibit significant inter-class overlap and insufficiently compact cluster structure, indicating limited discriminative power when relying solely on appearance and motion cues. In contrast, Figure 2b shows that CVTE produces more structured and compact clusters with clearer spacing. The results demonstrate that integrating these components enhances the discriminative power of the learned embeddings, making the distinction between categories clearer. Figures 2c and 2d show... Figure 2 d further highlights the role of knowledge distillation. Compared with CVTE, the distilled student model CVTE-S can form more compact clusters and reduce the overlap between easily confused categories, while maintaining a lightweight architecture. The teacher model CVTE-T exhibits the clearest cluster boundaries and the highest inter-class separation, demonstrating its stronger ability to capture subtle interaction differences. Overall, these visualization results are consistent with the quantitative results, indicating that feature discriminability shows a progressive improvement from the backbone model to CVTE, and then from CVTE-S to CVTE-T.

[0084] 5.2 Confusion Matrix Analysis This application uses a confusion matrix to evaluate the overall performance of the proposed model. As shown in Figure 3, the diagonal elements indicate that most categories achieve high per-category recall, with writing, reading_book, and sleeping achieving the highest results. In contrast, crying and shouting have relatively low recall rates, suggesting that these interaction- or emotion-related categories remain more challenging in the current setting.

[0085] Crying is most frequently predicted as get_angry, and reverse confusion (get_angry→crying) also exists, indicating bidirectional ambiguity in negative emotional behaviors. For interaction-related actions, shouting is sometimes mapped to yawning, while yawning is predicted as shouting or sleeping, consistent with common facial motor cues associated with mouth opening and low arousal states. For head-down movements, answering_questions are frequently confused with looking_at_phone and writing. Furthermore, when phone-related cues are weak or absent and fine-grained motor dynamics are limited, looking_at_phone is sometimes misclassified as sleeping or reading_book. Meanwhile, shaking_head is predicted as shouting and yawning, consistent with the co-occurrence of head movements in high-activation interaction segments. Future work will incorporate richer IoT modalities and longer-term temporal contexts to strengthen fine-grained facial and person-object evidence, thereby reducing residual confusion under real-world noise, occlusion, and domain bias.

[0086] 5.3 Running Time and Throughput Analysis To evaluate deployment efficiency, this application uses the same input settings (16 frames, ... 224×224 Inference latency and throughput were measured for four variants with the same inference configuration (batch size = 1); inference speed was reported in frame-level FPS (frames / s); Figure 4 summarizes the accuracy-speed trade-offs on CSBR10; the single-modal vision baseline covers a wider accuracy-FPS range due to differences in backbone complexity; while large-scale InternVideo2 maintains competitive accuracy but has significantly lower FPS; HCMT and CSBR improve Top-1 accuracy compared to vision-only models, but reduce throughput due to multimodal fusion overhead.

[0087] In contrast, CVTE-S is in a more favorable region in Figure 4, with a Top-1 accuracy higher than the existing classroom-specific multimodal baseline while maintaining a significantly higher FPS. The teacher model CVTE-T further improves accuracy but shifts to the left in the figure, consistent with the higher inference cost brought about by its larger capacity encoder. Overall, the distribution in Figure 4 shows that the proposed CVTE design can achieve a competitive balance between recognition accuracy and real-time throughput under controlled inference settings.

[0088] This application proposes CVTE for student behavior recognition in IoT-enabled smart classrooms. CVTE aligns synchronous video with contextual text into a shared embedding space and enhances multimodal representation learning through graph-based relation modeling, relation-driven description enhancement, and contextual conditional cues generated from IoT metadata. To support practical deployment, this application further introduces a teacher-student distillation scheme to transfer multimodal knowledge from a high-capacity teacher model to a lightweight student model. Experiments show that this method achieves a competitive tradeoff between accuracy and efficiency.

[0089] On CSBR10, the student model CVTE-S achieved a Top-1 accuracy of 86.57% at 71 GFLOPs and 56 FPS, outperforming existing multimodal classroom baselines and running with higher throughput. The teacher model CVTE-T further improved the Top-1 accuracy to 88.23%, corresponding to 282 GFLOPs and 32 FPS, providing a stronger upper bound for distillation performance. Further evaluations on UCF-101 and HMDB-51 show that the learned multimodal embeddings can be transferred from classroom scenarios to standard action recognition benchmarks. Future work will explore richer, more granular classroom signals, including additional IoT modalities and longer-range temporal context, to improve the model's robustness to real-world noise, occlusion, and domain shifts.

[0090] It should be noted that, in this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0091] This article uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only for the purpose of helping to understand the method and core ideas of the present invention. The above descriptions are only preferred embodiments of the present invention. It should be noted that due to the limitations of textual expression, while there are objectively infinite specific structures, those skilled in the art can make several improvements, modifications, or changes without departing from the principles of the present invention, and can also combine the above technical features in an appropriate manner. These improvements, modifications, changes, or combinations, or the direct application of the inventive concept and technical solution to other situations without modification, should all be considered within the scope of protection of the present invention.

Claims

1. A method for behavior recognition based on visual text contrast embedding and graph relational context fusion, characterized in that, Includes the following steps: Step 1: Contrastive visual-text embedding. Through contrastive representation learning, synchronized classroom videos and contextual text information are aligned into a shared embedding space. Step 2: Graph-based relational modeling. Building on Step 1, graph-based relational modeling is introduced to capture the dependencies between behavior categories and enhance multimodal representation. Step 3: Relationship-driven description enhancement. Building on step 2, relationship-driven description enhancement and contextual condition cues generated from IoT metadata are used to enrich the semantics of the text. Step 4: Based on Steps 1 to 3, construct a contrastive visual-text embedding model (teacher CVTE model) for IoT-enabled student behavior recognition. Step 5: Introduce a teacher-student knowledge distillation strategy to transfer the multimodal knowledge from the teacher CVTE model (high-capacity teacher model) in Step 4 to the lightweight student model (student CVTE model), thus constructing an end-to-end CVTE framework.

2. The method for visual text contrast embedding and graph relation context fusion for behavior recognition according to claim 1, characterized in that, The CVTE framework consists of three functional layers: a data acquisition layer, an edge layer, and a cloud layer. The data acquisition layer captures synchronous multimodal inputs from IoT sensors, including high-resolution classroom video streams and contextual text logs. The edge layer performs lightweight preprocessing, such as denoising and frame sampling, and performs real-time inference using a compact student CVTE model. The cloud layer receives inference output and selected feature summaries from the edge and performs large-scale analysis to improve the interpretability and long-term monitoring capabilities of the system. This layer aggregates time behavior patterns, supports time series analysis, and builds student behavior profiles. It is also the place for training and updating teacher CVTE models.

3. The method for visual text contrast embedding and graph relation context fusion for behavior recognition according to claim 2, characterized in that, The CVTE module employs a CLIP-style dual encoder architecture, further enhanced by graph-based relational modeling, relation-driven description enhancement, and contextual conditional hints, given a video segment. and its corresponding text description Student visual encoder and student text encoder The input is mapped to a shared feature space, and then passed through a projection head. and Get embedded: The components constitute a lightweight model deployed on resource-constrained IoT edge devices. During the training phase, CVTE also builds a high-capacity teacher network. This network has the same architectural design as the student model, but with greater width and depth, serving as a strong source of supervision for knowledge distillation.

4. The method for visual text contrast embedding and graph relation context fusion for behavior recognition according to claim 3, characterized in that, CVTE constructs a scene graph for each video frame or short clip. During training, regions of interest (ROIs) are pre-extracted offline and stored as lightweight annotations. During deployment, edge devices run only the compact CVTE-S backbone network for real-time inference, while YOLOv8 is used only in the offline preprocessing stage to generate a set of ROIs. These regions are projected as initial node embeddings: Build a relationship graph on top of these entities , among which the side Defined by spatial proximity or semantic prior; optional edge features Used to encode relative position or motion cues, the Graph Attention Network (GAT) then employs standard attention-based message passing, residual connections, and layer normalization. L Information is transmitted and exchanged within the layers. go through L After the layer, relationship-aware node features Aggregation is performed using the attention-based READOUT operator to obtain a single graph-level embedding: in and As learnable parameters, the final visual embedding incorporates global backbone features. Graph-level features Combining them: in Indicating splicing, the same graph modeling module is also instantiated into the teacher network, and relation enhancement visual features are generated for distillation. .

5. The method for visual text contrast embedding and graph relation context fusion for behavior recognition according to claim 4, characterized in that, CVTE introduces a relationship-driven description enhancement mechanism that generates auxiliary text descriptions from interaction graphs, using graph-level embedding. Used as a global semantic summary of classroom interactions, subsequently, a lightweight decoder by Generate relation descriptions for conditions : The generated descriptions are encoded using the same text encoder, resulting in... And as an additional text view for the corresponding visual fragment; for those containing N Small batches of fragments, Indicates the first The set of positive samples of text fragments is used to extend the visual-to-text contrast loss into a multi-positive-sample form: in Represents cosine similarity. Temperature parameter; symmetrical text-to-visual loss Similarly defined, the average of the two is denoted as the overall comparison target of the student network. The teacher network used the same contrast target. It can be trained or fine-tuned, and provides semantically rich multimodal representations for distillation.

6. The method for visual text contrast embedding and graph relation context fusion for behavior recognition according to claim 5, characterized in that, IoT-enabled classrooms can provide rich contextual metadata. CVTE uses signals to modulate the text encoder through contextual conditional cues, enabling... This represents the concatenated metadata vector, the context encoder. Will Mapped to One soft cue vector: in For the text embedding dimension, for lexicalized text sequences and its embedding The hint is added as an adaptive prefix to the front of the sequence: This is then fed into the Transformer-based text encoder in the student network to obtain context-aware text embeddings. The same prompting mechanism can also be applied to the teacher network, enabling both models to benefit from IoT-aware semantic conditions.

7. The method for visual text contrast embedding and graph relation context fusion for behavior recognition according to claim 6, characterized in that, To regularize contextual hints and avoid overfitting to specific classroom configurations, Frobenius norm penalty is applied: in Denotes the Frobenius norm. Control the regularization strength.

8. The method for visual text contrast embedding and graph relation context fusion for behavior recognition according to claim 7, characterized in that, The knowledge distillation scheme from teacher network to student network specifically involves: for the same input pair... Teacher model generates embeddings and corresponding category posterior The student model generates and ， At the prediction level, Kullback-Leibler divergence with temperature scaling is used: in The distillation temperature. Indicates temperature In addition, the feature layer distillation term is used to encourage the student model to simulate the multimodal representation of the teacher model: Finally, the overall training objective for the student network integrates contrastive loss, cue regularization, and distillation: in , and Control the relative contributions of different items.

Images