AI face emotion perception evaluation and analysis method and system

Abstract

The invention relates to the field of artificial intelligence, discloses an AI face emotion perception evaluation and analysis method and system, and aims to solve the problem that an existing student emotion perception system is insufficient in authenticity, continuity, multi-dimensional analysis and effective intervention and conflicts with students. The method comprises the steps that multi-modal student data such as face, voice, limbs and behavior logs are collected and preprocessed, and features are extracted; aggregated emotion vectors are generated through multi-head self-attention network fusion features; and analyzing the emotional state and combining the context to construct a personalized emotional portrait so as to generate an emotional evaluation report and an intervention suggestion. The system has the functions of multi-modal data acquisition, preprocessing, feature extraction, feature fusion, emotional state analysis, context fusion reasoning and report intervention suggestion generation. According to the technical scheme, the authenticity, continuity, accuracy and depth of emotion perception are improved, accurate intervention is provided, and the conflict psychology of students is effectively reduced.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, specifically relating to an AI-based method and system for facial emotion perception, assessment, and analysis. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, its application in affective computing is becoming increasingly widespread, particularly demonstrating enormous potential in mental health assessment, human-computer interaction optimization, and personalized service provision. As an important branch of AI, affective computing aims to enable computers to recognize, understand, express, and respond to human emotions, thereby achieving more natural and intelligent human-computer interaction. Among these technologies, facial emotion perception and analysis is a key technology in affective computing. By capturing and analyzing facial expression features to infer an individual's emotional state, it provides new perspectives and technical support for fields such as psychological assessment, educational guidance, and public safety.

[0003] Among them, the application of AI-powered facial emotion perception, assessment, and analysis systems in educational settings, especially in school mental health monitoring, has attracted widespread attention. These systems aim to effectively identify, assess, and track students' emotional states through intelligent analysis of their facial expressions, thereby assisting schools in promptly identifying potential psychological distress or emotional abnormalities, providing data support for psychological counseling and intervention, and ultimately promoting students' healthy growth and development.

[0004] In existing technologies, some school emotion perception and analysis systems adopt an offline, scenario-based data collection model. This involves inviting students to a private psychological analysis room, continuously filming their facial expressions using a fixed camera, and then analyzing these facial features based on a pre-set algorithm to generate emotion assessment results. However, this approach has significant limitations. First, the scenario lacks realism. Students in unfamiliar analysis rooms are prone to the "observational effect," causing their expressions to deviate from their true emotions, leading to distorted analysis results. Second, the time window is limited; short-term data collection cannot cover the dynamic changes in emotions, especially for introverted or observant students who may deliberately suppress their true state, preventing the system from capturing key signals. Furthermore, the analysis dimension is singular. The system mainly relies on facial expression data, lacking comprehensive analysis of various physiological signals such as voice tone and body movements, easily overlooking implicit cues closely related to emotions. At the application level, the results also have limited reference value. The generated reports are mostly basic emotion labels, lacking in-depth interpretation of emotion intensity and underlying causes, and failing to incorporate students' daily background information, making it difficult to provide psychological counselors with targeted intervention guidelines. Meanwhile, this passive participation model may create resistance among students, making them perceive entering the analysis room as a pressure, which in turn exacerbates the concealment of emotions and further reduces the effectiveness and reliability of the assessment. Therefore, how to overcome the shortcomings of existing technologies in terms of authenticity, comprehensiveness, multi-dimensional analysis, and application value, and to construct a more accurate, seamless, and insightful assessment and analysis method and system that can deeply understand students' emotional states, has become an urgent technical challenge to be solved. Summary of the Invention

[0005] This invention provides an AI-powered facial emotion perception assessment and analysis method and system, aiming to address the limitations of existing student emotion perception systems in terms of scene realism, time window, analysis dimensions, and result reference value, as well as the resulting student resistance. This allows for the natural, continuous, multimodal, and deeply contextualized accurate perception and assessment of student emotions.

[0006] According to one aspect of the present invention, an AI-based facial emotion perception assessment and analysis method is provided, comprising the following steps: Acquire multimodal raw student data, which includes: student facial image sequence data acquired by the image acquisition unit, student voice data acquired by the audio acquisition unit, student body movement sequence data acquired by the body movement acquisition unit, and student behavior log data acquired by the behavior log acquisition unit. The steps for obtaining multimodal raw student data specifically include: In the daily learning scenario of students, the image acquisition unit continuously acquires the video stream data of students at a preset frame rate, and extracts the student's facial image sequence data from the video stream data. The image acquisition unit is equipped with a dynamic lighting compensation module to cope with changes in lighting. The audio acquisition unit continuously acquires the student's voice signal at a preset sampling frequency, and performs sound source separation on the voice signal to extract the target student's speech data. The audio acquisition unit is equipped with an environmental noise reduction module to suppress background noise. The limb movement acquisition unit continuously acquires the student's three-dimensional skeleton posture data or limb key point sequence data at a preset data update rate. By using a pre-defined application programming interface or data integration module, students' online learning behavior, social interaction records, and background profile information can be periodically obtained from student learning management systems, social interaction platforms, or electronic archives to generate student behavior log data. The original multimodal student data is preprocessed, including timestamp synchronization, data alignment, missing value imputation, and outlier filtering, to form a multimodal time-series dataset in a unified format. Multimodal feature extraction is performed on the multimodal time-series dataset, and the multimodal feature extraction includes: Facial geometric features, facial motion unit features, and facial expression texture features are extracted from the facial image sequence data using a facial feature extraction model. The speech feature extraction model is used to extract pitch, volume, speech rate, intonation, and spectrogram features from the speech data; A limb movement feature extraction model is used to extract body posture features, gesture features, and micro-movement features from the limb movement sequence data; The behavioral log feature extraction model is used to extract learning engagement features, social activity features, and emotional tendency text features from the behavioral log data. Multimodal feature fusion is performed on the facial geometric features, facial action unit features, facial expression texture features, pitch, volume, speech rate, intonation, spectral features, body posture features, gesture features, micro-movement features, learning engagement features, social activity features, and emotional tendency text features to generate an aggregated emotion representation vector. The multimodal feature fusion adopts a deep neural network structure based on a multi-head self-attention mechanism. Its input is feature vectors from different modalities. The contribution of each modality to the aggregated emotional representation is dynamically adjusted through attention weights, and the cross-correlation information between modalities is captured to generate the aggregated emotional representation vector. Emotional state analysis is performed based on the aggregated emotion representation vector, and the emotion state analysis includes: The aggregated emotion representation vector is processed using an emotion classification and regression model to identify students' basic emotion categories and quantify emotion intensity. The basic emotion categories include positive emotions, negative emotions, and neutral emotions, and the emotion intensity is represented by continuous numerical values. The temporal changes of the aggregated emotion representation vector are analyzed using an emotion shift detection model to identify abrupt changes and durations of students' emotions. The emotion shift detection model is implemented based on a time series anomaly detection algorithm. A personalized emotional profile of a student is constructed using a context fusion and deep reasoning module. The steps for constructing the personalized emotional profile of a student include: Long-term temporal modeling units are used to model students' historical emotional state sequence data to capture the long-term evolution trend and periodic pattern of emotions. The long-term temporal modeling units are implemented based on temporal convolutional networks or long short-term memory networks. The background information embedding unit is used to convert structured background information such as students' academic performance, family environment and social relationships into embedding vectors, and then fuse them with the long-term emotional time-series features. By using an event correlation analysis engine to analyze teacher comments, student self-reports, or external event data, and combining them with the aforementioned emotional transition points, potential emotional triggering events and environmental factors can be identified. Based on the long-term emotional evolution trend, periodic patterns, emotional inflection points, triggering events, environmental factors, structured background information, and emotional intensity, the student's personalized emotional profile is generated. The personalized emotional profile includes an emotional baseline, emotional sensitivity points, emotional coping patterns, and potential risk factors. Based on the student's personalized emotional profile, an emotional assessment report and intervention recommendations are generated. The steps for generating the emotional assessment report and intervention recommendations include: Generate a visual emotion assessment report, which includes a time-series curve of students' emotions, annotation of key emotion events, evidence chains of each modality, and emotion risk level. The emotion risk level is divided into low risk, medium risk, and high risk according to a preset threshold. Based on the student's personalized emotional profile and emotional risk level, personalized psychological intervention strategies or learning environment adjustment suggestions are generated through an expert system or a reinforcement learning-based decision-making model. These strategies or suggestions are then directly provided to the school counselor or parents.

[0007] Furthermore, the facial feature extraction model employs facial keypoint detection, facial action unit recognition, and texture feature extraction algorithms based on convolutional neural networks. Facial keypoint detection is used to locate key areas of the student's face, facial action unit recognition is used to analyze subtle movements of facial muscles, and texture feature extraction is used to capture subtle changes on the skin surface.

[0008] Furthermore, the speech feature extraction model employs acoustic feature analysis and emotion recognition algorithms based on deep neural networks. The acoustic feature analysis is used to extract the physical properties of the speech signal, and the emotion recognition is used to determine the emotional information contained in the speech.

[0009] Furthermore, the limb movement feature extraction model employs pose estimation and micro-movement recognition algorithms based on three-dimensional convolutional neural networks or graph convolutional networks. The pose estimation is used to obtain the student's overall posture information, and the micro-movement recognition is used to capture subtle, unconscious movements of the student's hands, head, or body.

[0010] Furthermore, the behavior log feature extraction model employs text sentiment analysis based on a pre-trained language model and sequence pattern mining algorithms. The text sentiment analysis is used to parse the emotional tendency of the text content, and the sequence pattern mining is used to discover repetitive or abnormal patterns in student behavior.

[0011] According to another aspect of the present invention, an AI-based facial emotion perception, assessment, and analysis system is provided, comprising: The multimodal raw student data acquisition module is used to acquire multimodal raw student data, which includes: student facial image sequence data acquired by the image acquisition unit, student voice data acquired by the audio acquisition unit, student body movement sequence data acquired by the body movement acquisition unit, and student behavior log data acquired by the behavior log acquisition unit. The multimodal raw student data acquisition module specifically includes: An image acquisition unit is used to continuously acquire video stream data of students at a preset frame rate in students' daily learning scenarios, and extract student facial image sequence data from the video stream data. The image acquisition unit is equipped with a dynamic lighting compensation module to cope with changes in lighting. An audio acquisition unit is used to continuously acquire the student's sound signal at a preset sampling frequency, and to perform sound source separation on the sound signal to extract the target student's speech data. The audio acquisition unit is equipped with an environmental noise reduction module to suppress background noise. The limb motion acquisition unit is used to continuously acquire the student's three-dimensional skeleton posture data or limb key point sequence data at a preset data update rate. The behavior log collection unit is used to periodically acquire students' online learning behavior, social interaction records, and background profile information from the student learning management system, social interaction platform, or electronic archives through a preset application programming interface or data integration module, in order to generate student behavior log data. The data preprocessing module is used to preprocess the multimodal raw student data. The data preprocessing includes timestamp synchronization, data alignment, missing value imputation, and outlier filtering, thereby forming a multimodal time series dataset in a unified format. A multimodal feature extraction module is used to extract multimodal features from the multimodal time-series dataset; The multimodal feature extraction module specifically includes: The facial feature extraction submodule is used to extract facial geometric features, facial action unit features, and facial expression texture features from the facial image sequence data using a facial feature extraction model. The speech feature extraction submodule is used to extract pitch, volume, speech rate, intonation, and spectral features from the speech data using a speech feature extraction model; The limb movement feature extraction submodule is used to extract body posture features, gesture features, and micro-movement features from the limb movement sequence data using a limb movement feature extraction model. The behavior log feature extraction submodule is used to extract learning engagement features, social activity features, and emotional tendency text features from the behavior log data using the behavior log feature extraction model. The multimodal feature fusion module is used to perform multimodal feature fusion on the facial geometric features, facial action unit features, facial expression texture features, pitch, volume, speech rate, intonation, spectral features, body posture features, gesture features, micro-motion features, learning engagement features, social activity features, and emotional tendency text features to generate an aggregated emotion representation vector. The multimodal feature fusion module adopts a deep neural network structure based on a multi-head self-attention mechanism. Its input is feature vectors from different modalities. It dynamically adjusts the contribution of each modality to the aggregated emotion representation through attention weights and captures the cross-correlation information between modalities to generate the aggregated emotion representation vector. The emotion state analysis module is used to perform emotion state analysis based on the aggregated emotion representation vector. The emotion state analysis module specifically includes: The emotion classification and regression unit is used to process the aggregated emotion representation vector using an emotion classification and regression model to identify the student's basic emotion categories and quantify the emotion intensity. The basic emotion categories include positive emotions, negative emotions, and neutral emotions, and the emotion intensity is represented by continuous numerical values. The emotion shift detection unit is used to analyze the temporal changes of the aggregated emotion representation vector using an emotion shift detection model to identify the abrupt changes and duration of students' emotions. The emotion shift detection model is implemented based on a time series anomaly detection algorithm. The context fusion and deep reasoning module is used to build personalized emotional profiles of students; The context fusion and deep reasoning module specifically includes: The long-term time series modeling unit is used to model the historical emotional state sequence data of students in order to capture the long-term evolution trend and periodic pattern of emotions. The long-term time series modeling unit is implemented based on a temporal convolutional network or a long short-term memory network. The background information embedding unit is used to convert structured background information such as students' academic performance, family environment and social relationships into embedding vectors and fuse them with the long-term emotional time-series features. The event correlation analysis engine is used to analyze teacher comments, student self-reports, or external event data, and combine them with the emotional transition points to identify potential emotional triggering events and environmental factors. The personalized emotion profile building unit is used to generate the student's personalized emotion profile based on the long-term emotion evolution trend, periodic pattern, emotion mutation point, triggering event, environmental factors, structured background information and emotion intensity. The personalized emotion profile includes an emotion baseline, emotion sensitivity points, emotion coping patterns and potential risk factors. The emotion assessment report and intervention suggestion generation module is used to generate emotion assessment reports and intervention suggestions based on the student's personalized emotion profile. The emotion assessment report and intervention suggestion generation module specifically includes: A visual emotion assessment report generation unit is used to generate a visual emotion assessment report. The report includes a time-series curve of students' emotions, annotation of key emotion events, evidence chains of each modality, and emotion risk level. The emotion risk level is divided into low risk, medium risk, and high risk according to a preset threshold. The intervention suggestion generation unit is used to generate personalized psychological intervention strategies or learning environment adjustment suggestions based on the student's personalized emotional profile and emotional risk level, through an expert system or a decision model based on reinforcement learning. The strategies or suggestions are directly provided to the psychological teacher or parents.

[0012] Furthermore, the facial feature extraction submodule utilizes facial keypoint detection, facial action unit recognition, and texture feature extraction algorithms based on convolutional neural networks. Facial keypoint detection is used to locate key areas of the student's face, facial action unit recognition is used to analyze subtle movements of facial muscles, and texture feature extraction is used to capture subtle changes on the skin surface.

[0013] Furthermore, the speech feature extraction submodule utilizes acoustic feature analysis and emotion recognition algorithms based on deep neural networks. The acoustic feature analysis is used to extract the physical properties of the speech signal, and the emotion recognition is used to determine the emotional information contained in the speech.

[0014] Furthermore, the limb movement feature extraction submodule utilizes pose estimation and micro-movement recognition algorithms based on 3D convolutional neural networks or graph convolutional networks. The pose estimation is used to obtain the student's overall posture information, and the micro-movement recognition is used to capture subtle, unconscious movements of the student's hands, head, or body.

[0015] Furthermore, the behavior log feature extraction submodule utilizes text sentiment analysis based on a pre-trained language model and sequence pattern mining algorithms. The text sentiment analysis is used to parse the emotional tendency of the text content, and the sequence pattern mining is used to discover repetitive or abnormal patterns in student behavior.

[0016] Compared with the prior art, the advantages and positive effects of the present invention are as follows: This invention overcomes the lack of authenticity in existing "offline scenario-based data collection" models. By covertly and continuously collecting multimodal data in students' daily learning scenarios, it avoids unnatural emotional expressions caused by the "observation effect," thereby significantly improving the authenticity and effectiveness of emotional data. The system can capture students' genuine emotional expressions in a natural state, avoiding the emotional pretense caused by tension or restraint in a psychological analysis room.

[0017] This invention addresses the limitation of time windows in existing solutions. Through continuous long-term monitoring, this system can capture the dynamic changes, evolution trends, and periodic patterns of students' emotions. It is particularly effective in identifying the emotional signals of introverted or observant students, revealing potential emotional problems that are difficult to detect in a short period. This provides rich and comprehensive time-series data support for in-depth analysis of students' emotions.

[0018] This invention overcomes the limitations of existing solutions that rely on a single analytical dimension. The system innovatively integrates multimodal information such as facial expressions, voice tone, body language, and behavioral logs. Through multimodal feature fusion technology, particularly a deep neural network structure based on multi-head self-attention, it can comprehensively capture the integrated expression of emotions across different modalities, compensating for the deficiencies of single-modal information. For example, when a student's facial expression is calm but their voice tone or body language (such as frequent hand-rubbing) indicates tension, the system can comprehensively judge their true emotional state, improving the accuracy and robustness of emotion perception.

[0019] This invention enhances the reference value of emotion assessment results. The system not only identifies basic emotion categories and quantifies emotion intensity, but also constructs a personalized emotion profile through context fusion and deep reasoning modules, combining students' historical emotion sequences, academic background, family environment, social relationships, and triggering events. This profile provides an emotional baseline, sensitivity points, coping patterns, and potential risk factors, thereby generating in-depth emotion analysis reports and precise, actionable psychological intervention strategies or learning environment adjustment suggestions. This provides mental health teachers with targeted intervention guidelines, rather than simple emotion labels.

[0020] This invention effectively reduces students' resistance. Because it employs a covert, non-invasive data collection method, students are unaware of being specifically observed in their daily environment, eliminating the pressure of "passive participation." This allows the system to integrate more smoothly into students' daily learning and life, improving its acceptance and effectiveness in practical applications.

[0021] In summary, this invention provides a more realistic, comprehensive, in-depth, and personalized solution for the assessment and analysis of students' emotion perception through multimodal, continuous, and contextual fusion-based deep analysis. It significantly improves the accuracy, depth, and practicality of emotion perception and provides strong technical support for students' mental health. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the overall technical architecture of the AI ​​facial emotion perception assessment and analysis method and system proposed in this invention.

[0023] Figure 2 This is a schematic diagram of the core principle framework of multimodal feature fusion in this invention.

[0024] Figure 3 This is a flowchart illustrating the logical process of acquiring and preprocessing multimodal raw student data in this invention.

[0025] Figure 4 This is a flowchart illustrating the logical process of multimodal feature extraction in this invention.

[0026] Figure 5 This is a flowchart illustrating the logical process of constructing personalized emotional profiles for students using context fusion and deep reasoning in this invention. Detailed Implementation

[0027] In the following description, specific terminology and structures are used to illustrate embodiments of the invention for ease of understanding. However, these descriptions are not intended to limit the invention, and those skilled in the art will understand that various modifications and substitutions can be made without departing from the spirit and scope of the invention. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0028] See Figure 1-5 This invention proposes an artificial intelligence-based facial emotion perception assessment and analysis method and system. The invention aims to construct personalized emotion profiles for students through continuous, seamless multimodal data acquisition, combined with advanced feature extraction, fusion, and contextual reasoning technologies. This provides in-depth, continuous, and targeted emotion assessment and analysis services, effectively improving the authenticity, comprehensiveness, and practicality of emotion perception. This method overcomes the limitations of single-modal data or short-term observations by comprehensively analyzing multi-source data such as facial expressions, voice tone, and body movements in students' daily learning and life scenarios. Combined with their historical emotional performance and environmental context information, it deeply explores the underlying causes and dynamic patterns of emotions. The core of this method lies in constructing an intelligent analysis framework capable of adaptive learning and continuous iteration, providing psychological teachers with more comprehensive, accurate, and instructive decision support.

[0029] This invention proposes an artificial intelligence-based facial emotion perception assessment and analysis method, the process of which includes the following steps: S110, acquire and preprocess the raw data of multimodal students; As a specific embodiment of the present invention, this step aims to overcome the limitations of traditional single-modal, short-time sampling. By integrating multiple sensors, it achieves continuous and imperceptible acquisition of multi-dimensional physiological and behavioral signals of students in uncontrolled, natural states. Furthermore, it standardizes and improves the quality of this raw data to provide high-quality input for subsequent feature extraction. Specifically, this step includes the following sub-steps: First, a distributed multimodal sensor network is deployed in classrooms, libraries, dormitories, and other areas where students frequently engage in daily activities. The visual sensors utilize a wide-angle, high-resolution camera array to continuously capture video stream data of students' facial expressions, upper body postures, and hand micro-movements at a frame rate of 30 frames per second. The audio sensors employ a high-sensitivity directional microphone array to collect speech stream data of students during interactions and monologues at a sampling rate of 44 kHz, combined with environmental noise suppression algorithms to ensure speech quality. Depth sensors, such as structured light or time-of-flight sensors, acquire three-dimensional posture coordinates of students' skeletal structures at a frequency of 15 frames per second. Simultaneously, the system integrates with a student information management system interface to obtain static contextual data such as student registration information, class schedules, and class activity records.

[0030] Secondly, the massive amounts of heterogeneous raw data collected undergo timestamp synchronization and alignment. Due to differences in sampling frequencies and transmission delays among different sensors, all data streams must be synchronized via a global timestamp server to ensure accurate subsequent feature fusion and guarantee precise time-series correspondence between different modalities. For example, video frames, audio sampling points, depth skeleton data, and event logs are all bound to a unified timestamp. Through linear interpolation or nearest-neighbor interpolation algorithms, high-frequency data is downsampled or low-frequency data is upsampled to a common reference frequency, such as 10 Hz, forming a unified multimodal time-series dataset.

[0031] Next, noise reduction and quality enhancement are performed on the synchronized multimodal raw data. For video data, preprocessing operations such as image denoising, brightness equalization, and color correction are performed to reduce the impact of lighting changes and shadow occlusion on face recognition. When a face is partially occluded, image inpainting techniques based on generative adversarial networks are used for restoration to improve the recognizability of facial expressions. For audio data, adaptive noise cancellation and speech enhancement algorithms are used to remove background environmental noise and background noise, and speech endpoint detection is performed to separate valid human voice segments. For deep skeleton data, Kalman filtering is used to smooth joint coordinates and eliminate outliers caused by sensor jitter or momentary occlusion.

[0032] Finally, data normalization and standardization are performed. The pre-processed raw data, such as video pixel values, audio amplitude, and skeleton coordinates, are scaled to a specific numerical range, such as between zero and one or between -1 and one, to eliminate the impact of differences in the dimensions of data from different modalities on subsequent model training and to accelerate model convergence. For example, the pixel values ​​of video frames can be normalized to their maximum and minimum values, while the energy features of audio can be standardized to zero mean and unit variance. After these processes, the system generates a structurally consistent, high-quality, and time-synchronized multimodal student raw dataset, laying a solid foundation for subsequent in-depth analysis.

[0033] S120, extract multimodal features; As a specific embodiment of the present invention, this step aims to efficiently and accurately extract low-dimensional, highly discriminative features closely related to students' emotional states from preprocessed multimodal raw data, avoiding redundancy and noise interference in emotion recognition. By designing a dedicated deep learning feature extractor for each modality, unique emotional cues in different modalities can be captured. Specifically, this step includes the following sub-steps: First, facial expression feature extraction is performed. For the preprocessed video frame sequence, a facial expression feature extractor based on a convolutional neural network is employed. This extractor contains multiple convolutional layers, pooling layers, and batch normalization layers to learn and abstract visual patterns of facial regions layer by layer. The front end of the extractor focuses on capturing changes in key facial feature points (e.g., 106 points such as the corners of the eyes, mouth, and eyebrows), which constitute the basic units of facial movement. The back end uses an attention mechanism to focus on core areas of emotional expression, such as micro-expressions around the eyes, changes in lip shape, and the dynamics of forehead wrinkles. Finally, the extractor outputs a 512-dimensional facial expression feature vector, which encodes the dynamic information of the student's facial expression at the current moment. The feature extraction process can be represented by the following formula:

[0034] in, This represents the input of a preprocessed video frame image. This represents a facial expression feature extractor based on a convolutional neural network. This represents the output facial expression feature vector.

[0035] Secondly, speech emotion feature extraction is performed. For the preprocessed speech stream data, a speech emotion feature extractor based on a recurrent neural network is employed. This extractor first converts the raw audio signal into a Mel-frequency cepstral coefficient sequence, which effectively characterizes the spectral properties of speech. Subsequently, this sequence is input into a network layer composed of gated recurrent units to capture the temporal dependencies of the speech signal, such as the changing trends of prosodic features like speech rate, intonation, pitch, and loudness, as well as details of acoustic features like sound quality and intelligibility. Furthermore, the extractor incorporates an emotion dictionary matching module to perform preliminary emotional tendency analysis of the speech content. Finally, the extractor outputs a 384-dimensional speech emotion feature vector, which comprehensively reflects the emotional information in the student's speech.

[0036] Next, we extract emotional features from body movements. For the preprocessed deep skeleton data or video pose data, we employ a body movement emotional feature extractor based on a graph convolutional network. This extractor treats the student's joints (e.g., 25 key points such as the head, neck, shoulders, elbows, wrists, hips, knees, and ankles) as nodes in a graph, and skeletal connections as edges, constructing a human skeleton graph. Through multi-layer graph convolution operations, the extractor can capture the spatial relationships between joints, such as posture changes like crossed arms, forward leaning, and hand rubbing, as well as temporal dynamics such as joint movement speed and acceleration. These posture features are often closely related to emotional states such as tension, relaxation, and frustration. Finally, the extractor outputs a 512-dimensional body movement feature vector, which represents the emotional information contained in the student's body posture and movements.

[0037] Through the multimodal feature extraction process described above, the system can separate independent and complementary emotion discrimination features from video, audio, and depth data, laying a refined data foundation for subsequent deep fusion and emotion analysis.

[0038] S130, perform multimodal feature fusion; As a specific embodiment of the present invention, this step aims to effectively integrate independent emotional features extracted from different modalities, and construct a more comprehensive and robust student emotion representation by mining the intrinsic connections and complementary information between them. Traditional methods often use simple feature concatenation, but this ignores the differences in the contribution of different modalities to emotional expression and is difficult to handle asynchrony and conflict between modalities. This step introduces a cross-modal attention mechanism to achieve dynamic weighting and deep interaction of features, so as to generate fusion features with higher discriminative power. Specifically, this step includes the following sub-steps: First, feature dimension alignment and initial fusion are performed. Since facial expression feature vectors, speech emotion feature vectors, and body movement emotion feature vectors may have different dimensions, they first need to be mapped to the same high-dimensional shared feature space through independent linear transformation layers or non-linear mapping layers (such as a single-layer fully connected network), for example, unifying the dimensions to 512. Subsequently, these dimension-aligned feature vectors undergo a preliminary concatenation operation to form a coarse-grained multimodal joint feature vector.

[0039] Secondly, cross-modal attention weights are calculated. To capture the relative importance of different modalities in emotional expression, a multi-head self-attention mechanism and a cross-attention mechanism are employed. For the feature vector of each modality, an independent self-attention layer is first used to allow the internal elements to interact, enhancing the modality's expressive power. Subsequently, a cross-attention mechanism is introduced, where features of one modality (e.g., facial expressions) are used as the query vector, and features of other modalities (e.g., speech and body language) are used as the key and value vectors, respectively. Attention weights are obtained by calculating the dot product of the query and the key, followed by soft maximum normalization. These weights reflect the degree to which each modality's features contribute to the final emotion determination at the current moment. For example, when a student conceals their facial expressions, the weights of speech tone and body language may be dynamically increased. The calculation process of these attention weights can be expressed by the following formula:

[0040] in, This represents the query vector for the i-th modality. This represents the key vector of the j-th mode. The dimension of the key vector. This represents the attention weights that the i-th modality pays attention to the j-th modality.

[0041] Next, a fusion feature vector is generated. Using the calculated attention weights, the feature vectors of each modality are weighted and summed to generate a highly compressed and information-rich multimodal fusion feature vector. This fusion feature vector not only retains the unique information of each modality, but more importantly, it encodes the synergistic effects generated by intermodal interactions. For example, when a face displays a slight smile while the tone of voice is slightly tense, the fusion feature vector can capture this complex and contradictory emotional signal.

[0042] Finally, a multilayer perceptron is used to further nonlinearly transform and reduce the dimensionality of the fused feature vector, eliminating any redundancy that may arise during the fusion process, and mapping it to a final, lower-dimensional (e.g., 256) comprehensive emotion representation space. This final fused feature vector is a refined expression of the student's current emotional state, integrating visual, auditory, and postural information, providing a comprehensive and robust input for subsequent deep emotion inference.

[0043] S140, performs context fusion and deep reasoning; As a specific embodiment of the present invention, this step aims to go beyond the simple identification of instantaneous emotional signals, and to deeply analyze students' emotions within their environment, temporal context, and personal historical emotional evolution trajectory. By fusing multi-source contextual information and combining advanced temporal modeling and inference techniques, this step aims to reveal the underlying causes of emotions, predict emotional development trends, and establish personalized emotional evolution models for students, thereby providing more interpretable and predictive emotional assessments. Specifically, this step includes the following sub-steps: First, multi-source contextual information encoding is performed. The system acquires and encodes various types of contextual information. Temporal context includes specific time periods of the day (e.g., morning, afternoon, evening), weekdays (e.g., Monday, weekend), and holidays. Spatial context includes the specific environment in which the student is located (e.g., classroom, library, gymnasium), and physical environmental parameters such as noise level, light intensity, and population density are acquired through environmental sensors. Social context includes the student's current interaction partners (e.g., teacher, classmates, alone) and the content of their interactions. Personal history context includes the student's past emotional records, academic performance, and social activity logs. All these heterogeneous contextual information, such as time period labels, location codes, and interaction types, are converted into numerical vectors through an embedding layer and uniformly mapped to a feature space compatible with the fused feature vectors.

[0044] Secondly, a context-aware emotion sequence is constructed. The multimodal fusion feature vector generated in the previous step, combined with the currently encoded contextual information, is input into a temporal model based on an attention mechanism, such as a transformer network or a recurrent neural network with long short-term memory units. This temporal model can capture the long-term dependencies and dynamic changes in students' emotions. For example, by learning the anxiety tendencies that students have consistently exhibited in a specific course (such as math class), the model can better explain the currently observed mild tension, rather than simply classifying it as a random phenomenon. The self-attention mechanism in the transformer network allows the model to assign different weights to emotional states and contextual information from different past time points when processing the current emotional state, thereby achieving deep modeling of the emotion evolution path.

[0045] Secondly, in-depth reasoning about the causes of emotions is conducted. Using a trained temporal model, a deeper level of reasoning is performed on the current emotional state of students. This goes beyond simply identifying emotion categories; it involves exploring the potential triggers for these emotions. For example, when a student exhibits negative emotions after a specific social interaction, the model, combined with historical data, may infer a tendency towards social anxiety; when a student consistently shows frustration when facing assignments in a particular subject, the model may infer the underlying reasons for their difficulty in that subject. The reasoning process is achieved by analyzing the activation patterns and attention weight distribution within the model, integrating features and contextual information. The emotion reasoning process can be represented by the following formula:

[0046] in, This represents the student's emotional state at time t. This represents a sentiment inference model based on a transformer network. This represents the multimodal fusion feature vector at time t. This represents the context encoding vector at time t. It represents information about historical emotional sequences.

[0047] Finally, a personalized emotion evolution model for each student is generated. Based on long-term emotion data and contextual information, the system constructs a unique emotion evolution model for each student. This model not only depicts the student's baseline emotional state, common emotion patterns and their intensity, but also identifies typical triggers and coping mechanisms that lead to emotional fluctuations. For example, the model might reveal that a student tends to exhibit high anxiety the night before an exam, but can quickly regain composure after receiving support from classmates. This personalized model is dynamically updated, continuously optimized and refined as new data is added, providing a solid basis for subsequent emotion profiling and intervention strategies.

[0048] S150: Construct personalized emotional profiles of students and output assessment and analysis results; As a specific embodiment of the present invention, this step is the final presentation and application stage of the entire method. It aims to transform the complex emotional information acquired and inferred in the preceding steps into an intuitive, comprehensive, and instructive personalized emotional profile, and output a structured assessment and analysis report. This not only provides quantitative indicators of emotion, but more importantly, it reveals the deep mechanisms and potential needs behind emotions, providing scientific decision support for school psychologists. Specifically, this step includes the following sub-steps: First, emotion classification and regression results are generated. Based on the emotional states obtained through deep inference, the system utilizes a multi-task learning model for emotion classification and regression. The emotion classification task categorizes student emotions into discrete labels, such as eight basic emotions: "joy," "calm," "surprise," "sadness," "anger," "fear," "disgust," and "anxiety," or more granular complex emotions, such as "curiosity," "frustration," and "confidence." The emotion regression task quantifies the intensity of each emotion, typically represented by continuous values ​​from zero to one, for example, anxiety intensity is 0.85. This multi-task model achieves simultaneous prediction of emotion category and intensity by sharing a low-level feature extractor and connecting it separately to the classification head and regression head.

[0049] Secondly, the system constructs personalized emotional profiles for students. It integrates emotion classification and regression results across multiple time scales with contextually inferred causes and evolutionary paths to generate personalized emotional profiles for each student. These profiles include key components: a long-term emotional baseline reflecting the student's average emotional tendency under normal conditions; short-term emotional fluctuations capturing emotional anomalies during specific events or periods; a list of emotion triggers identifying external stimuli or internal factors that cause recurring specific emotions; an emotional resilience index quantifying the student's speed and ability to recover from negative emotions; and emotional expression preferences, such as a tendency towards internalization or externalization. The profiles are further enhanced in depth and breadth by linking them to information such as the student's academic performance and social networks.

[0050] Next, a structured assessment and analysis report is generated. Based on the personalized emotional profile, the system automatically generates a detailed and structured emotional assessment and analysis report. The report includes: an overview of the student's current emotional state and a comparison with historical trends; analysis of the frequency, duration, and intensity of specific negative emotions (such as anxiety and depression); identification of specific situations or events that may lead to emotional fluctuations; early warning of potential mental health risks (such as an assessment of the risk of depression in cases of prolonged low mood); and targeted intervention recommendations, such as recommendations for psychological counseling, suggestions for adjusting learning strategies, and provision of emotion regulation techniques. The report is written in an objective and professional style, avoiding the use of vague or evaluative terms.

[0051] Finally, the system provides real-time alerts and a visual interface. In addition to regularly generated reports, the system can provide real-time alerts based on set thresholds for drastic changes or long-term abnormal trends in student emotions. A visual dashboard presents emotion data, profiles, and analysis reports to school counselors in an intuitive and easy-to-understand manner. The visual interface supports multi-dimensional filtering and time-series backtracking, allowing teachers to quickly locate and understand students' emotional problems, enabling timely intervention and personalized psychological support and guidance.

[0052] This invention proposes an artificial intelligence facial emotion perception, assessment, and analysis system, the structure of which includes: The multimodal raw student data acquisition module is used to acquire multimodal raw student data, which includes: student facial image sequence data acquired by the image acquisition unit, student voice data acquired by the audio acquisition unit, student body movement sequence data acquired by the body movement acquisition unit, and student behavior log data acquired by the behavior log acquisition unit. The multimodal raw student data acquisition module specifically includes: An image acquisition unit is used to continuously acquire video stream data of students at a preset frame rate in students' daily learning scenarios, and extract student facial image sequence data from the video stream data. The image acquisition unit is equipped with a dynamic lighting compensation module to cope with changes in lighting. An audio acquisition unit is used to continuously acquire the student's sound signal at a preset sampling frequency, and to perform sound source separation on the sound signal to extract the target student's speech data. The audio acquisition unit is equipped with an environmental noise reduction module to suppress background noise. The limb motion acquisition unit is used to continuously acquire the student's three-dimensional skeleton posture data or limb key point sequence data at a preset data update rate. The behavior log collection unit is used to periodically acquire students' online learning behavior, social interaction records, and background profile information from the student learning management system, social interaction platform, or electronic archives through a preset application programming interface or data integration module, in order to generate student behavior log data. The data preprocessing module is used to preprocess the multimodal raw student data. The data preprocessing includes timestamp synchronization, data alignment, missing value imputation, and outlier filtering, thereby forming a multimodal time series dataset in a unified format. A multimodal feature extraction module is used to extract multimodal features from the multimodal time-series dataset; The multimodal feature extraction module specifically includes: The facial feature extraction submodule is used to extract facial geometric features, facial action unit features, and facial expression texture features from the facial image sequence data using a facial feature extraction model. The speech feature extraction submodule is used to extract pitch, volume, speech rate, intonation, and spectral features from the speech data using a speech feature extraction model; The limb movement feature extraction submodule is used to extract body posture features, gesture features, and micro-movement features from the limb movement sequence data using a limb movement feature extraction model. The behavior log feature extraction submodule is used to extract learning engagement features, social activity features, and emotional tendency text features from the behavior log data using the behavior log feature extraction model. The multimodal feature fusion module is used to perform multimodal feature fusion on the facial geometric features, facial action unit features, facial expression texture features, pitch, volume, speech rate, intonation, spectral features, body posture features, gesture features, micro-motion features, learning engagement features, social activity features, and emotional tendency text features to generate an aggregated emotion representation vector. The multimodal feature fusion module adopts a deep neural network structure based on a multi-head self-attention mechanism. Its input is feature vectors from different modalities. It dynamically adjusts the contribution of each modality to the aggregated emotion representation through attention weights and captures the cross-correlation information between modalities to generate the aggregated emotion representation vector. The emotion state analysis module is used to perform emotion state analysis based on the aggregated emotion representation vector. The emotion state analysis module specifically includes: The emotion classification and regression unit is used to process the aggregated emotion representation vector using an emotion classification and regression model to identify the student's basic emotion categories and quantify the emotion intensity. The basic emotion categories include positive emotions, negative emotions, and neutral emotions, and the emotion intensity is represented by continuous numerical values. The emotion shift detection unit is used to analyze the temporal changes of the aggregated emotion representation vector using an emotion shift detection model to identify the abrupt changes and duration of students' emotions. The emotion shift detection model is implemented based on a time series anomaly detection algorithm. The context fusion and deep reasoning module is used to build personalized emotional profiles of students; The context fusion and deep reasoning module specifically includes: The long-term time series modeling unit is used to model the historical emotional state sequence data of students in order to capture the long-term evolution trend and periodic pattern of emotions. The long-term time series modeling unit is implemented based on a temporal convolutional network or a long short-term memory network. The background information embedding unit is used to convert structured background information such as students' academic performance, family environment and social relationships into embedding vectors and fuse them with the long-term emotional time-series features. The event correlation analysis engine is used to analyze teacher comments, student self-reports, or external event data, and combine them with the emotional transition points to identify potential emotional triggering events and environmental factors. The personalized emotion profile building unit is used to generate the student's personalized emotion profile based on the long-term emotion evolution trend, periodic pattern, emotion mutation point, triggering event, environmental factors, structured background information and emotion intensity. The personalized emotion profile includes an emotion baseline, emotion sensitivity points, emotion coping patterns and potential risk factors. The emotion assessment report and intervention suggestion generation module is used to generate emotion assessment reports and intervention suggestions based on the student's personalized emotion profile. The emotion assessment report and intervention suggestion generation module specifically includes: A visual emotion assessment report generation unit is used to generate a visual emotion assessment report. The report includes a time-series curve of students' emotions, annotation of key emotion events, evidence chains of each modality, and emotion risk level. The emotion risk level is divided into low risk, medium risk, and high risk according to a preset threshold. The intervention suggestion generation unit is used to generate personalized psychological intervention strategies or learning environment adjustment suggestions based on the student's personalized emotional profile and emotional risk level, through an expert system or a decision model based on reinforcement learning. The strategies or suggestions are directly provided to the psychological teacher or parents.

[0053] Furthermore, the facial feature extraction submodule utilizes facial keypoint detection, facial action unit recognition, and texture feature extraction algorithms based on convolutional neural networks. Facial keypoint detection is used to locate key areas of the student's face, facial action unit recognition is used to analyze subtle movements of facial muscles, and texture feature extraction is used to capture subtle changes on the skin surface.

[0054] Furthermore, the speech feature extraction submodule utilizes acoustic feature analysis and emotion recognition algorithms based on deep neural networks. The acoustic feature analysis is used to extract the physical properties of the speech signal, and the emotion recognition is used to determine the emotional information contained in the speech.

[0055] Furthermore, the limb movement feature extraction submodule utilizes pose estimation and micro-movement recognition algorithms based on 3D convolutional neural networks or graph convolutional networks. The pose estimation is used to obtain the student's overall posture information, and the micro-movement recognition is used to capture subtle, unconscious movements of the student's hands, head, or body.

[0056] Furthermore, the behavior log feature extraction submodule utilizes text sentiment analysis based on a pre-trained language model and sequence pattern mining algorithms. The text sentiment analysis is used to parse the emotional tendency of the text content, and the sequence pattern mining is used to discover repetitive or abnormal patterns in student behavior.

[0057] The various modules of the aforementioned system work collaboratively to achieve comprehensive, continuous, and in-depth perception, assessment, and analysis of students' emotions. Data is collected seamlessly through a distributed sensor network, deep learning models extract and fuse multimodal features, and deep reasoning is performed in conjunction with contextual information to ultimately construct personalized emotion profiles and provide professional assessment reports and intervention suggestions. This approach not only significantly improves the authenticity and accuracy of emotion perception but also provides a more scientific and refined support tool for school psychological counseling.

[0058] The above are merely specific embodiments of the present invention, but the technical features of the present invention are not limited thereto. Any simple changes, equivalent substitutions, or modifications made based on the present invention to solve essentially the same technical problems and achieve essentially the same technical effects are all covered within the protection scope of the present invention.

Claims

1. An AI-based facial emotion perception, assessment, and analysis method, characterized in that, include: Acquire multimodal raw student data, which includes: student facial image sequence data acquired by the image acquisition unit, student voice data acquired by the audio acquisition unit, student body movement sequence data acquired by the body movement acquisition unit, and student behavior log data acquired by the behavior log acquisition unit. The original multimodal student data is subjected to timestamp synchronization, data alignment, missing value imputation, and outlier filtering to form a multimodal time-series dataset in a unified format. Multimodal feature extraction is performed on the multimodal time-series dataset. This multimodal feature extraction includes: extracting facial geometric features, facial action unit features, and facial expression texture features from the facial image sequence data using a facial feature extraction model; extracting pitch, volume, speech rate, intonation, and spectrogram features from the speech data using a speech feature extraction model; extracting body posture features, gesture features, and micro-movement features from the body action sequence data using a body action feature extraction model; and extracting learning engagement features, social activity features, and emotionally charged text features from the behavior log data using a behavior log feature extraction model. Multimodal feature fusion is performed on the facial geometric features, facial action unit features, facial expression texture features, pitch, volume, speech rate, intonation, spectral features, body posture features, gesture features, micro-movement features, learning engagement features, social activity features, and emotional tendency text features to generate an aggregated emotion representation vector. Emotional state analysis is performed based on the aggregated emotion representation vector. The emotion state analysis includes identifying the student's basic emotion categories and quantifying the emotion intensity, as well as analyzing the temporal changes of the aggregated emotion representation vector to identify the abrupt changes and duration of the student's emotions. A personalized emotional profile of students is constructed through a context fusion and deep reasoning module. The steps of constructing a personalized emotional profile of students include modeling the students' historical emotional state sequence data to capture the long-term evolution trend and periodic pattern of emotions, converting the students' structured background information into embedded vectors and fusing them with the long-term emotional time sequence features, and analyzing event data to identify potential emotional triggering events and environmental factors. Based on the student's personalized emotional profile, an emotional assessment report and intervention suggestions are generated. The steps for generating the emotional assessment report and intervention suggestions include generating a visual emotional assessment report and generating personalized psychological intervention strategies or learning environment adjustment suggestions.

2. The AI ​​facial emotion perception assessment and analysis method according to claim 1, characterized in that, The steps for obtaining multimodal raw student data include: In the daily learning scenario of students, the image acquisition unit continuously acquires the video stream data of students at a preset frame rate, and extracts the student's facial image sequence data from the video stream data. The image acquisition unit is equipped with a dynamic lighting compensation module to cope with changes in lighting. The audio acquisition unit continuously acquires the student's voice signal at a preset sampling frequency, and performs sound source separation on the voice signal to extract the target student's speech data. The audio acquisition unit is equipped with an environmental noise reduction module to suppress background noise. The limb movement acquisition unit continuously acquires the student's three-dimensional skeleton posture data or limb key point sequence data at a preset data update rate. By using a pre-defined application programming interface or data integration module, the system periodically retrieves students' online learning behavior, social interaction records, and background profile information from student learning management systems, social interaction platforms, or electronic archives to generate the student behavior log data.

3. The AI ​​facial emotion perception assessment and analysis method according to claim 2, characterized in that, The steps of performing timestamp synchronization, data alignment, missing value imputation, and outlier filtering on the multimodal raw student data include: The massive heterogeneous raw data collected is time-stamped and aligned to ensure that different modal data correspond precisely on the time axis. The synchronized multimodal raw data is subjected to noise reduction and quality enhancement, which includes performing image denoising, brightness equalization, color correction, adaptive noise cancellation, and Kalman filtering to smooth joint coordinates. The data after noise cancellation and quality enhancement are normalized and standardized to uniformly scale to a specific numerical range.

4. The AI ​​facial emotion perception assessment and analysis method according to claim 3, characterized in that, The facial feature extraction model employs facial key point detection, facial action unit recognition, and texture feature extraction algorithms based on convolutional neural networks. The speech feature extraction model employs acoustic feature analysis and emotion recognition algorithms based on deep neural networks; The limb movement feature extraction model employs a pose estimation and micro-movement recognition algorithm based on a three-dimensional convolutional neural network or a graph convolutional network. The behavior log feature extraction model employs text sentiment analysis and sequence pattern mining algorithms based on pre-trained language models.

5. The AI ​​facial emotion perception assessment and analysis method according to claim 4, characterized in that, The facial landmark detection is used to locate key areas on the student's face. The facial motion unit identifies and analyzes minute movements of facial muscles. The texture feature extraction is used to capture subtle changes on the skin surface.

6. The AI ​​facial emotion perception assessment and analysis method according to claim 5, characterized in that, The multimodal feature fusion adopts a deep neural network structure based on a multi-head self-attention mechanism. Its input is feature vectors from different modalities. The contribution of each modality to the aggregated emotional representation is dynamically adjusted through attention weights, and the cross-correlation information between modalities is captured to generate the aggregated emotional representation vector. The multimodal feature fusion also includes mapping feature vectors of different modalities to the same high-dimensional shared feature space through independent linear transformation layers or nonlinear mapping layers, followed by preliminary splicing operations to form a coarse-grained multimodal joint feature vector. The fused feature vector is nonlinearly transformed and dimensionality reduced by a multilayer perceptron to eliminate redundancy that may occur during the fusion process, and then mapped to the emotion comprehensive representation space.

7. The AI ​​facial emotion perception assessment and analysis method according to claim 6, characterized in that, The process of identifying students' basic emotion categories and quantifying emotion intensity includes processing the aggregated emotion representation vector using an emotion classification and regression model to identify students' basic emotion categories and quantify emotion intensity. The basic emotion categories include positive emotions, negative emotions, and neutral emotions, and the emotion intensity is represented by continuous numerical values. The analysis of the temporal changes of the aggregated emotion representation vector to identify abrupt changes and durations of student emotions includes using an emotion shift detection model to analyze the temporal changes of the aggregated emotion representation vector. The emotion shift detection model is implemented based on a time series anomaly detection algorithm.

8. The AI ​​facial emotion perception assessment and analysis method according to claim 7, characterized in that, The steps for constructing personalized emotional profiles of students include: Long-term temporal modeling units are used to model students' historical emotional state sequence data to capture the long-term evolution trend and periodic pattern of emotions. The long-term temporal modeling units are implemented based on temporal convolutional networks or long short-term memory networks. The background information embedding unit is used to convert structured background information such as students' academic performance, family environment and social relationships into embedding vectors, and then fuse them with the long-term emotional time-series features. By using an event correlation analysis engine to analyze teacher comments, student self-reports, or external event data, and combining them with the aforementioned emotional transition points, potential emotional triggering events and environmental factors can be identified. Based on the long-term emotional evolution trend, periodic patterns, emotional inflection points, triggering events, environmental factors, structured background information, and emotional intensity, a personalized emotional profile of the student is generated. The personalized emotional profile includes an emotional baseline, emotional sensitivity points, emotional coping patterns, and potential risk factors.

9. An AI-powered facial emotion perception, assessment, and analysis system, characterized in that, include: The multimodal raw student data acquisition module is used to acquire multimodal raw student data, which includes student facial image sequence data acquired by the image acquisition unit, student voice data acquired by the audio acquisition unit, student body movement sequence data acquired by the body movement acquisition unit, and student behavior log data acquired by the behavior log acquisition unit. The data preprocessing module is used to perform timestamp synchronization, data alignment, missing value imputation, and outlier filtering on the multimodal raw student data, thereby forming a multimodal time-series dataset in a unified format. A multimodal feature extraction module is used to extract multimodal features from the multimodal time-series dataset. The multimodal feature extraction includes extracting facial geometric features, facial action unit features, and facial expression texture features from the facial image sequence data; and extracting pitch, volume, speech rate, intonation, and spectrogram features from the speech data. Body posture features, gesture features, and micro-movement features are extracted from the limb movement sequence data; learning engagement features, social activity features, and emotional text features are extracted from the behavior log data. The multimodal feature fusion module is used to perform multimodal feature fusion on the facial geometric features, facial action unit features, facial expression texture features, pitch, volume, speech rate, intonation, spectral features, body posture features, gesture features, micro-motion features, learning engagement features, social activity features, and emotional tendency text features to generate an aggregated emotion representation vector. The emotion state analysis module is used to perform emotion state analysis based on the aggregated emotion representation vector. The emotion state analysis includes identifying the student's basic emotion categories and quantifying the emotion intensity, as well as analyzing the temporal changes of the aggregated emotion representation vector to identify the abrupt changes and duration of the student's emotions. The context fusion and deep reasoning module is used to construct personalized emotional profiles of students. The construction of personalized emotional profiles of students includes modeling the historical emotional state sequence data of students to capture the long-term evolution trend and periodic pattern of emotions, converting the structured background information of students into embedded vectors and fusing them with the long-term emotional time sequence features, and analyzing event data to identify potential emotional triggering events and environmental factors. The emotion assessment report and intervention suggestion generation module is used to generate emotion assessment reports and intervention suggestions based on the student's personalized emotion profile. The generation of emotion assessment reports and intervention suggestions includes generating a visual emotion assessment report and generating personalized psychological intervention strategies or learning environment adjustment suggestions.

10. The AI ​​facial emotion perception, assessment, and analysis system according to claim 9, characterized in that, The multimodal raw student data acquisition module includes: An image acquisition unit is used to continuously acquire video stream data of students at a preset frame rate in students' daily learning scenarios, and extract student facial image sequence data from the video stream data. The image acquisition unit is equipped with a dynamic lighting compensation module to cope with changes in lighting. An audio acquisition unit is used to continuously acquire the student's sound signal at a preset sampling frequency, and to perform sound source separation on the sound signal to extract the target student's speech data. The audio acquisition unit is equipped with an environmental noise reduction module to suppress background noise. The limb motion acquisition unit is used to continuously acquire the student's three-dimensional skeleton posture data or limb key point sequence data at a preset data update rate. The behavior log collection unit is used to periodically acquire students' online learning behavior, social interaction records, and background profile information from the student learning management system, social interaction platform, or electronic archives through a preset application programming interface or data integration module, so as to generate the student behavior log data.

Images