A Student Emotion Recognition Method and System Based on LLaVA Multimodal Model

Abstract

This invention relates to a student emotion recognition method and system based on the LLaVA multimodal model, and relates to the fields of deep learning and emotion recognition technology. It addresses the problems of existing multimodal models lacking dynamic resolution strategies and failing to optimize for small targets, leading to unstable recognition accuracy in real classroom environments. The system includes a multimodal data acquisition module for acquiring raw data from a camera module and a physiological sensor module; a dynamic resolution adjustment module for adjusting the camera module's resolution based on classroom activity type and image complexity; a multimodal fusion module for mapping raw data to a shared subspace to generate fused features; and a lightweight model inference module for inference and recognition, outputting the recognition result. This invention utilizes real-time, hardware-aligned private data to adapt to environmental changes, resulting in superior robustness in real-world scenarios compared to existing technologies.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and emotion recognition technology, specifically to a student emotion recognition method and system based on the LLaVA multimodal model. Background Technology

[0002] Currently, emotion recognition technology based on computer vision and multimodal learning has been applied in the education field, but existing technical solutions (such as MIT's Picard framework and Stanford's attention mechanism model) have significant technical shortcomings when implemented in real classrooms:

[0003] The emotion recognition objectives lack educational relevance and granularity: Most existing emotion recognition systems are categorized based on six basic emotions defined in psychology (such as happiness, sadness, anger, and fear). However, these emotion categories fail to effectively reflect students' cognitive state and learning engagement in the highly dynamic and complex classroom environment. For example, when students encounter learning obstacles, they are more likely to exhibit "confusion" or "frustration" than extreme "anger"; they may display "contemptuous" micro-expressions when they disagree with the textbook content. Existing models perform extremely poorly in distinguishing between "confusion" and key teaching emotions such as "neutrality" or "frustration," resulting in a severe disconnect between the analysis results and actual learning states, failing to provide teachers with effective intervention guidelines.

[0004] The sheer number of model parameters is severely mismatched with edge computing power: Current mainstream multimodal large models typically have billions of parameters, while the low-cost edge devices to be deployed in this system (such as Raspberry Pi 4B) are only equipped with basic ARM Cortex-A72 CPUs. Existing technology lacks a mature solution for efficiently deploying 7B-level multimodal models on such resource-constrained devices and achieving online inference on the edge. Direct deployment would easily lead to excessively high inference latency, failing to meet the real-time monitoring needs of the classroom, resulting in high technology implementation costs and substandard performance.

[0005] Lack of data generalization validation mechanisms: Most existing studies rely solely on labeled datasets collected in laboratories or single institutions for training and testing, lacking validation mechanisms on unlabeled data streams collected in real classroom environments with uneven lighting and diverse student behaviors. This results in poor model robustness and an inability to effectively address the domain shift problem between "laboratory data" and "real classroom data."

[0006] In summary, this invention provides a student emotion recognition method and system based on the LLaVA multimodal model to solve the aforementioned technical problems. Summary of the Invention

[0007] To address the problems of existing multimodal models not involving dynamic resolution strategies and being unable to optimize for small targets, resulting in unstable recognition accuracy in real classroom environments, this invention provides a student emotion recognition method and system based on the LLaVA multimodal model.

[0008] A student emotion recognition method based on the LLaVA multimodal model is implemented through the following steps:

[0009] Step 1: Construct a dual-track dataset;

[0010] The standardized, publicly available DIPSER dataset was used as the training set; real-time acquired image frames and physiological data were used as the test set.

[0011] Step 2: Multimodal feature fusion;

[0012] Feature extraction is performed on the image frames and physiological data in the test set respectively, and fused features are generated; specifically:

[0013] Image frames are input into the CLIP visual encoder to extract deep visual features. Physiological data is input into a 1D-CNN encoder to extract physiological features. ;

[0014] A weighted fusion formula is used to combine visual features. and physiological characteristics Mapping to a shared subspace generates a fused feature M, expressed as follows:

[0015] ;

[0016] In the formula, This is the weight matrix. For activation function, and These are visual weights and physiological weights, respectively.

[0017] Step 3: Load the pre-trained emotion perception knowledge distillation model; the model is pre-trained offline using the training set described in Step 1, based on a teacher-student architecture; the fusion features generated in Step 2 are input into the pre-trained student model for lightweight inference testing, outputting a nine-dimensional probability distribution vector, and the maximum probability is taken as the final recognition result.

[0018] The present invention also provides a student emotion recognition system based on the LLaVA multimodal model, which includes a multimodal data acquisition module, a dynamic resolution adjustment module, a multimodal fusion module, and a lightweight model inference module;

[0019] The multimodal data acquisition module is used to acquire the raw data collected by the camera module and the physiological sensor module;

[0020] The dynamic resolution adjustment module is used to adjust the resolution of the camera module according to the type of classroom activity and the complexity of the image;

[0021] The multimodal fusion module is used to map the original data to a shared subspace to generate fusion features; the fusion features are then used for inference and recognition through a lightweight model inference module to output the recognition results.

[0022] The beneficial effects of this invention are:

[0023] 1. The emotion recognition system of this invention constructs a system for visual motion analysis, GPIO signal triggering, and I... 2 The system uses a closed-loop control link written to C hardware registers. Instead of relying on software interpolation, the system calculates motion intensity (threshold 0.22 / 0.31) using an optical flow algorithm and sends I / O signals directly to the IMX477 camera via Raspberry Pi GPIO pins. 2 C commands switch the sensor output mode (336×336 @ 15fps / 560×560 @ 10fps) in real time at the hardware level.

[0024] 2. The emotion recognition system described in this invention completely resolves the contradiction between "always high resolution causing computing power crashes" and "always low resolution resulting in loss of small target details" in edge devices. In static teaching scenarios, this strategy reduces redundant computing overhead by 37%, ensuring that the Raspberry Pi 4B does not overheat or reduce its frequency during long-term operation; in dynamic scenarios, the effective pixel density is increased by 2.8 times, significantly improving the ability to capture micro-expressions of students in the back row or in occluded scenarios.

[0025] 3. The emotion recognition system described in this invention uses a hardware-level timestamp synchronization circuit based on PWM pulses. It abandons the traditional soft synchronization scheme that relies on operating system time and forces the physical alignment of visual exposure and physiological sampling at the nanosecond level. This compresses the time deviation of multimodal data from hundreds of milliseconds to <5ms, effectively solving the data misalignment problem caused by system scheduling delay.

[0026] 4. The emotion recognition system described in this invention adopts a lightweight edge deployment structure integrating in-memory computing. Targeting the characteristics of the Raspberry Pi 4B's ARM Cortex-A72 architecture, a three-level optimized inference path is designed: "structural pruning + INT8 quantization + CMSIS-NN operator acceleration." Combined with knowledge distillation technology, knowledge from a large 7B model is transferred to a lightweight student model, compressing the model size to 1 / 4 of its original size. The multimodal perception algorithm was successfully run on a general-purpose development board with only 13W power consumption. The inference latency was controlled within a second-level time window (depending on the resolution mode) to meet the needs of teaching intervention, achieving high-performance AI edge computing on low-cost hardware.

[0027] 5. The emotion recognition system described in this invention achieves fine-grained perception of teaching emotions, breaking through the limitations of the traditional six basic emotion categories and realizing the recognition of nine emotions (including neutral, happy, sad, angry, fearful, surprised, disgusted, contemptuous, and confused) as defined by the DIPSER standard. Specifically, through a weighted loss function, feature enhancement is applied to the key teaching emotions "confusion" (weight × 1.5) and "contempt" (weight × 1.5). This allows for the identification of deeper learning states such as "not understanding" (confusion) or "passive resistance" (contempt), rather than simply categorizing it as "unhappy," significantly improving the accuracy of educational feedback.

[0028] 6. The emotion recognition method described in this invention adopts a dual-track data-driven mechanism, proposing a dual-track strategy of "basic training on a public dataset + real-time prediction on a private dataset". It utilizes standard public datasets to endow the model with general knowledge, while using real-time collected and hardware-aligned private data to adapt to environmental changes. This solves the domain shift problem of the model between different classroom lighting conditions and different student groups, making the system more robust in real-world scenarios than competing products trained only in laboratory environments.

[0029] 7. The emotion recognition method described in this invention employs an emotion perception knowledge distillation framework; that is, it adopts a "teacher-student" architecture, transferring the high-order semantic understanding capabilities of the large LLaVA-13B model to the edge small model. By using a joint loss function, the student model is forced to imitate the feature extraction logic of the teacher model for difficult samples; while significantly reducing the computational power requirement, it retains more than 88% of the micro-expression understanding capabilities of the teacher model, achieving "small model, great wisdom". Attached Figure Description

[0030] Figure 1 This is a flowchart of the student emotion recognition method based on the LLaVA multimodal model described in this invention;

[0031] Figure 2 A flowchart for reasoning using a student model;

[0032] Figure 3 This is a block diagram of the student emotion recognition system based on the LLaVA multimodal model described in this invention;

[0033] Figure 4 This is a comparison diagram showing the effect of the method of the present invention compared with existing methods. Detailed Implementation

[0034] Specific Implementation Method 1: Combination Figure 1 and Figure 2 This embodiment describes a student emotion recognition method based on the LLaVA multimodal model, which is implemented through the following steps:

[0035] Step S1. Construct a dual-track dataset; use the standardized public dataset DIPSER as the training set; use the field data collected in real time by the multimodal data acquisition module as the test set;

[0036] During the training phase: The standardized public dataset DIPSER was used as the training source for the teacher model. This dataset contains labeled triples of "facial image-physiological data-emotion label", with labels covering 9 key emotions (including neutral, confused, contemptuous, etc.) to ensure that the model has general multimodal feature extraction capabilities.

[0037] During the testing phase: Real-time data acquisition was conducted using a multimodal data acquisition module. Specifically, RGB image streams were acquired via a Raspberry Pi HQ Camera, and physiological signals were acquired at a frequency of 10Hz or 15Hz via a physiological sensor module (integrating PPG heart rate, pressure, and body temperature sensors).

[0038] In this embodiment, a "private dataset" (which has a hardware timestamp and a data structure isomorphic to the training set) is collected in real time using a classroom-end camera module and a physiological sensor module. This dataset serves as the input source for the model to perform real-time inference, effectively improving the robustness of the model in real classroom scenarios.

[0039] In this implementation, the system abandons the soft synchronization scheme that relies on the operating system time, and uses a Raspberry Pi PWM module to generate a high-precision pulse signal as a global clock source; the rising edge of the PWM signal simultaneously triggers the frame synchronization pin of the camera and the data interrupt pin of the physiological sensor; and a uniform nanosecond-level timestamp is added to each frame of image and each physiological data packet. The specific process is as follows:

[0040] The Raspberry Pi reads RGB image frames via the CSI interface and simultaneously reads physiological data such as heart rate (HRV low-frequency power), stress value, and body temperature via USB. A PWM module is used to map the physiological data (10Hz / 15Hz) onto the corresponding image frames. This allows the time deviation of multimodal data to be controlled within 5ms, with an accuracy of 0.01 seconds, thereby constructing a private test set isomorphic to the training set.

[0041] Step S2. Task-aware dynamic resolution adjustment;

[0042] The OpenCV algorithm is used to calculate the mean optical flow difference between image frames in real time, and the type of classroom activity is determined based on the mean optical flow difference.

[0043] In this embodiment, a closed-loop control strategy is adopted:

[0044] Static scene determination: When the average optical flow difference is detected to be ≤0.22 (e.g., a teacher lecturing, students sitting quietly), the system determines it as a static scene. The dynamic resolution adjustment module outputs a low level via GPIO, instructing the camera to switch to a low-load mode of 336×336@15fps, establishing a low-latency video stream buffer. This strategy, while ensuring optical flow detection sensitivity, reduces redundant computational overhead by 37%, ensuring that the device does not overheat or reduce its frequency during long-term operation.

[0045] Dynamic Scene Determination: When the average optical flow difference is detected to be ≥0.31 (e.g., during group discussions, students frequently turning their heads), the system determines it to be a dynamic scene. The GPIO outputs a high level, which is then transmitted via I... 2 The C command forces the camera to instantly switch to a high-precision mode of 560×560@10fps, providing a high-resolution image source for subsequent keyframe sampling. This strategy effectively increases pixel density by 2.8 times, significantly improving the ability to capture micro-expressions of students in the back row or those whose faces are obscured.

[0046] This embodiment also includes a transition interval preservation strategy;

[0047] When the average optical flow difference is detected to be between 0.22 and 0.31, the system determines that the current scene state has not changed significantly, and the dynamic resolution adjustment module does not output a switching command, maintaining the resolution mode of the previous moment. This hysteresis interval is designed to prevent the jitter phenomenon caused by frequent switching of camera resolution due to small fluctuations in optical flow values ​​near the critical point, thus ensuring the stability of the system.

[0048] In this implementation, a task-aware dynamic resolution strategy is designed to adaptively adjust the camera input resolution (336×336 / 560×560) based on the type of classroom activity and the intensity of image motion. While ensuring the clarity of nine key emotional features, this effectively reduces redundant computational overhead and ensures stable operation of the Raspberry Pi 4B under high load.

[0049] Step S3. Multimodal feature fusion;

[0050] An asynchronous sampling strategy is employed. The multimodal data acquisition module continuously writes real-time data into the video stream circular buffer of the edge computing host. The multimodal fusion module then reads the latest image frame (keyframe) and its corresponding physiological data from this buffer in a non-blocking manner. The extracted keyframe visual and physiological feature vectors are then processed by a CLIP encoder and a 1D-CNN model, respectively, to generate fused features. Finally, using a fusion strategy with different weights, the visual and physiological features are mapped to a unified semantic space. Specifically:

[0051] The image frames captured by the camera module are input into the CLIP visual encoder to extract deep visual features with a dimension of 512. ;

[0052] Physiological data such as heart rate (HRV low-frequency power), pressure value, and body temperature collected by the physiological sensor module are input into a 1D-CNN encoder to extract physiological features with a dimension of 256. ;

[0053] Adaptive weighted fusion: Employing a weighted fusion formula to combine visual features and physiological characteristics Mapping to a shared subspace to generate fused features The formula is as follows:

[0054] ;

[0055] in, This is the weight matrix. This is the activation function. The weight allocation logic is dynamically adjusted based on the scenario determination.

[0056] In static mode: Settings (Visual weight) (Physiological weighting). At this point, the focus is on capturing attention through facial expressions.

[0057] In dynamic mode: Settings (Visual weight) (Physiological weighting). At this point, the focus is on using physiological signals (such as an increase in heart rate due to intense discussion) to assist in recognition and compensate for the blurring of visual information caused by facial occlusion or movement.

[0058] Step S4. Load the pre-trained emotion perception knowledge distillation model; the model is pre-trained offline using the training set described in Step 1, based on a teacher-student architecture; such as Figure 2 As shown, the fusion features generated in step two are input into the pre-trained student model for lightweight inference testing, and a nine-dimensional probability distribution vector is output. The maximum probability value is taken as the final recognition result.

[0059] In this embodiment, a "teacher-student" joint training strategy is adopted to compress the reasoning ability of a large-scale model into a lightweight edge model. The teacher model adopts the LLaVA-13B model, and the LLaVA-13B model is trained offline using the training set in step S1 to generate nine probability distributions for emotions, namely: soft labels;

[0060] The student model uses LLaVA-7B and is trained using soft labels and a training set. During training, the parameters of the student model are in an active state and are updated in real time through the backpropagation algorithm to obtain the final trained student model.

[0061] In this implementation, the teacher and student models define a guidance relationship based on probability distribution alignment: the teacher model uses an LLaVA-13B model loaded with general multimodal pre-trained weights. Throughout the distillation training process, its weight parameters remain frozen and do not participate in gradient updates. The student model uses an LLaVA-7B model; during training, the student model's parameters are in an active state and are updated in real-time via backpropagation.

[0062] In this embodiment, the training set is only used for the offline training phase on the server side. The multimodal data in the training set is simultaneously input into the teacher model and the student model. For the same batch of data, the teacher model uses it as a benchmark to calculate the probability distribution (soft label) of the sample in 9 emotions. The student model makes predictions on the data at the same time, and its prediction results are subject to the dual constraints and corrections of the real label and the soft label.

[0063] The real-time data collected on-site is used as a test set only for online inference on the Raspberry Pi edge computing host. The unlabeled data collected on-site is only input into the deployed student model. At this time, the teacher model has been removed, and the student model independently uses the trained parameters to perform real-time emotion classification on the on-site data and output the final result.

[0064] In this embodiment, a joint loss function comprising weighted classification loss and knowledge distillation loss is constructed. .

[0065] Weighted classification loss function To ensure the student model's ability to fit the ground truth, class weights are introduced to address the issue of key emotions being easily overlooked in teaching scenarios. :

[0066] ;

[0067] In the formula: This represents an index of 9 emotion categories; The true labels provided for the training set (one-hot encoded, with values ​​of 0 or 1); The normalized probability values ​​output by the student model; These are the category weight coefficients. The strategy is set as follows: for the two categories "confused" and "contemptuous", set... For the remaining 7 types of emotions, set... This weighting setting forces the model to prioritize correcting prediction errors for key teaching emotions during backpropagation.

[0068] Knowledge distillation loss function The similarity between the output probability distributions of the teacher model and the student model is measured using KL divergence.

[0069] ;

[0070] In the formula: and These are the unnormalized prediction scores of the teacher model and the student model before Softmax activation, respectively. The distillation temperature is set in this embodiment. Higher temperature values ​​can smooth the probability distribution, thereby amplifying the probability values ​​of non-target categories, which is more conducive to student model learning. This refers to the Softmax function.

[0071] Total loss function Ultimately used to update student model parameters:

[0072] ;

[0073] In the formula: As a balance coefficient, in this embodiment, it is set to... This indicates that supervision by real-labeled personnel is just as important as guidance from teacher-modeled personnel. The term is used to balance the magnitude of the gradient, ensuring that the two parts of the loss are numerically on the same scale.

[0074] Step S5. Deploy the student model trained in step S4 at the edge.

[0075] In this implementation, a three-level optimization strategy is adopted: To ensure that the trained student model can achieve real-time inference on the Raspberry Pi 4B (ARM Cortex-A72), the system implements the following three-level optimization path during the model export and deployment phase:

[0076] Level 1: Structural pruning: During the student model export stage, redundant channels and attention heads that contribute little to emotion recognition are removed, and the model size is physically compressed while preserving the core feature extraction capability of the visual encoder.

[0077] Level 2: INT8 / INT4 Hybrid Quantization: Utilizing post-training quantization techniques, the precision of the student model weights is reduced from 32-bit floating-point numbers to 8-bit integers. For some non-sensitive layers, INT4 quantization is further attempted. This significantly reduces memory access bandwidth pressure with almost no loss of recognition accuracy.

[0078] Level 3: CMSIS-NN Operator Acceleration: Targeting the ARM architecture characteristics of the Raspberry Pi, the CMSIS-NN kernel library is invoked. This library optimizes convolution, fully connected, and matrix multiplication operations through the SIMD (Single Instruction Multiple Data) instruction set, replacing general-purpose inefficient operators and significantly improving the computational throughput at the edge.

[0079] Specific Implementation Method Two: Combination Figure 3 This embodiment describes the student emotion recognition method based on the LLaVA multimodal model described in Specific Embodiment 1. This system achieves fine-grained multimodal emotion recognition of nine key emotions (neutral, happy, sad, angry, fearful, surprised, disgusted, contemptuous, and confused) on a low-computing Raspberry Pi 4B platform through hardware and software co-optimization.

[0080] The system described in this embodiment includes a multimodal data acquisition module, a dynamic resolution adjustment module, a multimodal fusion module, and a lightweight model inference module;

[0081] The multimodal data acquisition module is used to acquire the raw data collected by the camera module and the physiological sensor module;

[0082] In this embodiment, the multimodal data acquisition module is used for dual-track data flow management; on the one hand, it loads the DIPSER public dataset (containing 9 emotion labels) for model training; on the other hand, during the inference stage, it collects data from the camera module and the physiological sensor module in real time to construct an unlabeled private prediction dataset.

[0083] The multimodal fusion module is used to map the original data to a shared subspace to generate fused features;

[0084] In this embodiment, the multimodal fusion module includes a visual encoder (CLIP) and a physiological signal encoder (1D-CNN), which are used to map multi-source data to a unified semantic space and perform feature concatenation to generate fused features;

[0085] The lightweight model inference module runs the LLaVA-7B student model generated based on knowledge distillation, which can identify nine emotions, including "confusion" and "contempt", and output probability distributions; the fused features are inferred and identified through the LLaVA-7B student model, and the identification results are output.

[0086] The dynamic resolution adjustment module adjusts the camera resolution according to the type of classroom activity and image complexity;

[0087] In this embodiment, the dynamic resolution adjustment module incorporates an optical flow algorithm based on OpenCV, which determines the classroom status (static lecture / dynamic discussion) based on changes in optical flow in the image, and outputs control signals to adjust the camera resolution.

[0088] This embodiment also includes an edge deployment optimization module, which integrates the INT8 quantization engine and the CMSIS-NN operator library to perform low-level acceleration for the Raspberry Pi CPU instruction set, ensuring efficient edge inference under low power consumption.

[0089] The system described in this embodiment is implemented through the following hardware: a camera, a physiological sensor, an edge computing host, a dynamic resolution controller, a timestamp synchronization circuit, and a result display module;

[0090] The camera uses a Raspberry Pi HQ Camera (based on a Sony IMX477 sensor), equipped with a 6mm wide-angle lens, and connects to the motherboard via a CSI interface, supporting hardware-level, zero-latency switching between 336×336 and 560×560 resolutions.

[0091] The physiological sensor integrates a photoplethysmography (PPG) heart rate sensor, a thin-film pressure sensor, and a non-contact infrared body temperature sensor. It aggregates data via a USB interface and supports a synchronous sampling rate of over 10Hz.

[0092] The edge computing host uses a Raspberry Pi 4B (8GB LPDDR4 RAM version), equipped with a quad-core Cortex-A72 processor (1.5GHz), serving as the system's control and computing core, hosting the Linux system and AI inference environment; it exchanges memory data via an internal AXI bus. Specifically, to adapt to the computing power characteristics of the edge, the edge computing host has partitioned an independent video stream circular buffer in the system memory. This buffer serves as an asynchronous decoupling interface between the multimodal data acquisition module and the lightweight model inference module, employing a "write-to-overwrite" strategy to store the latest N frames (N=5) of raw images and aligned physiological data, thus resolving the rate mismatch between high-frequency data acquisition and deep model inference. By modifying the underlying overvoltage parameters and optimizing heat dissipation, the overall power consumption is kept below 13W while ensuring full-speed operation at the 1.5GHz clock frequency; the final results are output to the result display module via a Micro-HDMI interface.

[0093] The dynamic resolution controller is based on Raspberry Pi GPIO pin communication. 2 The C control unit is used to send resolution switching commands to the camera.

[0094] In this embodiment, the dynamic resolution controller monitors the software layer for instructions in real time. When it receives a "high resolution" instruction (GPIO high level), it uses I... 2 Configure the IMX477 sensor register on the C bus to switch its output mode to 560×560@10fps; when a "low resolution" instruction (GPIO low level) is received, switch to 336×336@15fps.

[0095] The timestamp synchronization circuit is a hardware signal generator implemented based on the PWM (Pulse Width Modulation) module of the Raspberry Pi 4B motherboard. It is used to generate synchronization pulses with nanosecond precision to force the alignment of image frames and sensor data.

[0096] In this embodiment, the timestamp synchronization circuit utilizes a Raspberry Pi PWM module to generate a high-precision pulse signal as a global clock source. This circuit assigns a uniform nanosecond-level timestamp to each frame of image and each physiological data packet, with the hardware-level synchronization error strictly controlled within 5ms and the timestamp accuracy reaching 0.01 seconds.

[0097] The result display module is connected to a display terminal via a Micro-HDMI interface, or a network interface module for data backhaul.

[0098] In this embodiment, the camera (IMX477) is directly connected to the CSI interface of the Raspberry Pi 4B motherboard via a CSI-2 (Camera Serial Interface) cable to ensure high-bandwidth image transmission. The physiological sensor is connected to the Raspberry Pi 4B motherboard via a USB 3.0 / 2.0 interface, and data transmission uses an asynchronous serial communication protocol. The dynamic resolution controller is connected to the GPIO pins of the Raspberry Pi (I... 2 The C SDA / SCL) timestamp synchronization circuit is connected to the Raspberry Pi's PWM output pin. The PWM signal simultaneously triggers the camera shutter and sensor sampling interrupts, achieving hardware-level alignment.

[0099] Specific Implementation Method Three: Combination Figure 4 This embodiment describes a student emotion recognition system based on the LLaVA multimodal model, as described in Specific Embodiment Two: This embodiment designs static lecture scenarios and dynamic discussion scenarios;

[0100] Scenario 1: Static lecture scenario;

[0101] Conditions: The teacher explains the core knowledge points at the podium, the students sit upright in their seats and listen attentively, with minimal head movement (movement intensity ≤ 0.22), and the classroom lighting is stable.

[0102] System operating status;

[0103] Resolution control: When the dynamic resolution adjustment module detects low motion intensity, the GPIO outputs a low level, instructing the camera to operate in a low-load mode of 336×336@15fps.

[0104] Multimodal weights: visual feature weights are set to 0.6, and physiological feature weights are set to 0.4 (at this point, it mainly depends on the focus of facial expression capture).

[0105] Edge computing power: The Raspberry Pi CPU load is maintained at around 60%, and the total power consumption is about 8W.

[0106] Recognition performance;

[0107] Response speed: Sub-second / second response time (meets the needs of teaching observation in static scenarios). Core emotion recognition: Accurately captures "confusion" (slightly furrowed brows, indicating misunderstanding) and "neutral" (eye contact).

[0108] Accuracy: 84.3%.

[0109] Scenario 2: Dynamic discussion scenario;

[0110] Conditions: Students are divided into groups to discuss the topic, leaning forward, frequently turning their heads to communicate with their peers (exercise intensity ≥ 0.31), and even their faces are obscured by each other.

[0111] System operating status;

[0112] Resolution control: When motion intensity detection triggers a threshold, the GPIO outputs a high level, instructing the camera to instantly switch to a 560×560@10fps high-precision mode to capture more pixel details.

[0113] Multimodal weights: Visual feature weights are automatically reduced to 0.4, while physiological feature weights are increased to 0.6.

[0114] Edge computing power: When the Raspberry Pi starts the INT8 deep analysis mode, the CPU load rises to 95% and the power consumption of the whole machine is about 12.5W.

[0115] Recognition performance;

[0116] Response speed: Within the effective teaching feedback time threshold. Core emotions to identify: Focus on identifying "happiness" (intense discussion, increased heart rate) and "contempt" (disdain for group members' opinions).

[0117] Accuracy: 78.6%.

[0118] like Figure 4 As shown in Table 1, Table 1 presents the results of comparing the existing methods with the method of the present invention.

[0119] Table 1

[0120]

[0121] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A student emotion recognition method based on an LLaVA multi-modal model, characterized in that: The method is realized by the following steps: Step one, constructing a double-track data set; The standardized public DIPSER data set is taken as the training set; the real-time collected image frames and physiological data are taken as the test set; Step two, multi-modal feature fusion; The image frames and physiological data in the test set are respectively subjected to feature extraction, and fusion features are generated; Specifically: inputting the image frame into a CLIP visual encoder to extract deep visual features ; Inputting the physiological data into a 1D-CNN encoder to extract physiological features ; The visual features and the physiological features are mapped to a shared subspace by using a weighted fusion formula to generate a fusion feature M. The visual features and the physiological features are mapped to a shared subspace by using a weighted fusion formula to generate a fusion feature M.​ It is expressed by the following formula: ； In the formula, is a weight matrix, is an activation function, and respectively are visual weight and physiological weight; the visual weight and the physiological weight are dynamically adjusted according to the scene of judgment; Step three, loading a pre-trained emotion perception knowledge distillation model; the model is pre-trained by the training set in step one, and is trained offline based on a teacher-student architecture; the fusion features generated in step two are input into the pre-trained student model for lightweight inference test, and a nine-dimensional probability distribution vector is output, and the maximum probability value is taken as the final recognition result; The method further includes a task perception-based dynamic resolution adjustment step; This step is used for real-time calculation of the optical flow difference mean value between image frames, and judgment of the classroom activity type according to the optical flow difference mean value; the classroom activity type is divided into a static scene and a dynamic scene.

2. The student emotion recognition method based on the LLaVA multi-modal model according to claim 1, characterized in that: When the optical flow difference mean value is detected to be ≤0.22, the static scene is judged; When the optical flow difference mean value is detected to be ≥0.31, the dynamic scene is judged.

3. The student emotion recognition method based on the LLaVA multi-modal model according to claim 1, characterized in that: In static scenes: Set , ; In dynamic scenarios: Set , .

4. The student emotion recognition method based on the LLaVA multi-modal model according to claim 1, characterized in that: In step one, a PWM module of a Raspberry Pi is used to generate a high-precision pulse signal as a hardware global clock source; the rising edge of the pulse signal simultaneously triggers a frame synchronization pin of a camera and a data interrupt pin of a physiological sensor, realizing hardware-level physical alignment of image frame collection and physiological data sampling, and generating a unified timestamp based on the alignment signal.

5. The student emotion recognition method based on the LLaVA multi-modal model according to claim 1, characterized in that: In step three, a teacher model and a student model are constructed by using knowledge distillation; the teacher model adopts LLaVA-13B, and the student model adopts LLaVA-7B; A joint loss function of a weighted classification loss function and a knowledge distillation loss function training the student model; The weighted classification loss function is expressed by the following equation: ； In the formula: is an index of nine emotional categories, is a category weight coefficient, is a true label provided for the training set, is a normalized probability value output by the student model; The knowledge distillation loss function is calculated using the KL divergence and is expressed by the following equation: ； wherein: and are the unnormalized prediction scores before the Softmax activation for the teacher and student models, respectively, is the distillation temperature, is the Softmax function; Total loss function for updating the student model parameters, expressed as: ； In the formula: is a balancing factor, for the magnitude of the balancing gradient.

6. A student emotion recognition system based on LLaVA multimodal model, characterized by: The system is used to realize the student emotion recognition method according to any one of claims 1 to 5; the system includes a multi-modal data collection module, a dynamic resolution adjustment module, a multi-modal fusion module and a lightweight model inference module; The multi-modal data collection module is used to acquire original data collected by a camera module and a physiological sensor module; The dynamic resolution adjustment module is used to adjust the resolution of the camera module according to the classroom activity type and the image complexity; The multi-modal fusion module is used to map the original data to a shared subspace to generate fusion features; the fusion features are subjected to inference recognition by the lightweight model inference module, and an identification result is output.

7. The LLaVA multimodal model based student emotion recognition system according to claim 6, characterized in that: The camera module and the physiological sensor module are both connected with a Raspberry Pi 4B mainboard; the dynamic resolution adjustment module is connected with a GPIO pin of the Raspberry Pi 4B mainboard; a timestamp synchronization circuit is connected with a PWM output pin of the Raspberry Pi; a PWM signal simultaneously triggers a shutter of the camera module and a sampling interrupt of the physiological sensor module, realizing hardware-level alignment.

8. The LLaVA multimodal model based student emotion recognition system according to claim 6, wherein: The hardware of the system includes a Raspberry Pi 4B mainboard, a dynamic resolution controller and a timestamp synchronization circuit; The dynamic resolution controller is an I 2 A C control unit is configured to send a resolution switching instruction to the camera module. The timestamp synchronization circuit is a hardware signal generator realized by a PWM module of the Raspberry Pi 4B mainboard, and is used to generate a nanosecond-level precision synchronization pulse signal, realizing alignment of image frames and sensor data; The core processor of the Raspberry Pi 4B mainboard is an edge computing host, which divides an independent video stream ring buffer in the system memory, the buffer serving as an asynchronous decoupling interface between a multi-modal data acquisition module and a lightweight model inference module, and adopting a write-overwrite strategy to store the latest preset number of image frames and aligned physiological data.

9. The LLaVA multimodal model based student emotion recognition system according to claim 8, characterized in that: In a static scene, the dynamic resolution controller outputs a low level through a GPIO pin to control the camera to switch to a low-load mode of 336*336@15fps. In a dynamic scene, the dynamic resolution controller outputs a high level through the GPIO pin, and outputs a low level through the I 2 The C instruction controls the camera to switch to a high-precision mode of 560x560@10fps.

10. The LLaVA multimodal model based student emotion recognition system according to claim 8, wherein: An edge deployment optimization module is further included, which integrates an INT8 quantization engine and a CMSIS-NN kernel library to accelerate the Raspberry Pi CPU instruction set at a bottom level.

Images