Audio-visual integrated immersive beauty cabin based on facial emotion recognition driving

Abstract

The invention relates to the technical field of emotion recognition and recommendation, in particular to an audio-visual integrated immersive beauty cabin based on facial emotion recognition drive, which comprises a student client, an administrator client and a server, the student client is used for collecting facial expression pictures, physiological signals and behavior data of students, identifying current emotional states of the students according to the facial expression pictures, the physiological signals and the behavior data, and interacting with the students; the server side is used for screening audio and video contents from a beauty database according to the current emotional state; and the administrator client is used for managing the beauty cabin, the beauty database and the student emotional state data by an administrator. According to the invention, personalized beauty content recommendation can be provided according to the emotional state and behavior characteristics of the student.

Description

Technical Field

[0001] The present invention relates to the field of emotion recognition and recommendation technology, and in particular to an audio-visual integrated immersive aesthetic education cabin driven by facial emotion recognition. Background Art

[0002] At present, due to various subjective and objective reasons, the proportion of students with negative psychological problems remains high, which brings great challenges to education and teaching management, students' daily life and family education. In this regard, although all kinds of schools have psychological counseling rooms and are actively promoting mental health counseling, this kind of conversational intervention requires a large number of professional psychological counselors, and students need a certain amount of courage to enter the psychological counseling room, so the positive guidance effect on students is limited.

[0003] Traditional psychological counseling is often conducted in a face-to-face interpersonal information exchange mode. Without establishing mutual trust, it is difficult for students to open up, so it often takes a long time to achieve the desired effect. With the development of artificial intelligence technology, especially in the advancement of large models and big data technology, it is possible to provide students with a solitary relaxation scene for personalized learning and emotional experience. How to create an immersive scene, effectively combine audio-visual perception with art therapy, and then classify students' emotional states, and timely intervene and positively guide students' negative emotions, is still a thorny problem that needs to be solved.

[0004] First, emotions play a vital role in the learning process. Studies have shown that students' emotional states directly affect their behavioral motivations, daily lives, and spiritual worlds. Therefore, it is of great significance to accurately identify students' emotional states, adjust audio-visual content according to these states, and guide their emotional experiences. Although current emotion recognition technology has made progress in many fields, its application in the field of education faces unique challenges. For example, students' emotional expressions in educational environments may be affected by a variety of factors, making emotion recognition more complex and difficult to achieve high accuracy. In addition, these technologies often need to process large data sets, and the real-time requirements put higher demands on data processing speed.

[0005] Secondly, people's core perception of beauty is mainly focused on the visual and auditory levels. Classical paintings and music highly condense the artist's emotional expression of beauty. In addition, the purpose of aesthetic education is not only to impart knowledge and artistic creation skills, but more importantly to stimulate students' creativity, improve their artistic literacy and aesthetic ability, and enrich their emotional expression. Therefore, aesthetic education needs to pay more attention to students' personalized development and emotional experience, requiring educators to provide customized art aesthetic content based on students' personalized emotional state and personality preferences, so that students can meet their aesthetic needs in a state of physical and mental relaxation. At present, personalized push systems are quite mature in recommending music, movies or shopping products, but applying these systems to the field of education, especially combining emotion recognition technology to recommend audio-visual content that can match students' emotions and have a positive impact, is still an open research field. Most existing recommendation systems rely on static user preferences or historical data, and lack the ability to respond to users' current emotional state in real time, reflecting a weak matching degree.

[0006] Thirdly, the art therapy theory of emotional intervention across the two fields of art and psychology is completely in line with human instinct and talent. Art education can effectively guide and stimulate people's inner experience and emotional expression. It is also a way of emotional and psychological expression in addition to language, and even beyond language communication. This immersive aesthetic education experience system constructed by audio-visual integrated art uses the beauty of art to change people's inner world, and immediately plays a positive or active role in guiding people's negative or negative emotions, improving students' mental health and achieving the effect of aesthetic experience.

[0007] In addition, with the development of information technology, multimedia and Internet technologies have been widely used in the field of education, providing new tools and platforms for aesthetic education. Especially in the school environment, using technology to create immersive and interactive learning experiences has become an important direction of educational innovation. However, how to effectively integrate these technologies, especially how to use these technologies to achieve real-time recognition and response to students' emotional states, remains a technical challenge.

[0008] To sum up, there is an urgent need to develop an immersive aesthetic education experience system that spans the two major fields of art and psychology and can combine emotion recognition and personalized recommendation technologies. Summary of the invention

[0009] The purpose of the present invention is to provide an audio-visual integrated immersive aesthetic education cabin driven by facial emotion recognition, which can provide personalized aesthetic education content recommendations based on students' emotional state and behavioral characteristics, and at the same time improve students' learning experience and effects through a comfortable learning environment and intuitive interactive interface.

[0010] To achieve the above object, the present invention provides the following solutions:

[0011] An audio-visual integrated immersive aesthetic education cabin based on facial emotion recognition, including: a student client, an administrator client and a server;

[0012] The student client is used to collect facial expression pictures, physiological signals and behavioral data of students, identify the current emotional state of students based on the facial expression pictures, physiological signals and behavioral data, and interact with students;

[0013] The server side is used to filter out audio and video content from the aesthetic education database according to the current emotional state;

[0014] The administrator client is used for the administrator to manage the aesthetic education cabin, the aesthetic education database and the students' emotional state data.

[0015] Optionally, the student client includes a data collection module, an emotion recognition module, a content playback module and an information interaction module;

[0016] The data collection module is used to collect the facial expression pictures, physiological signals and behavioral data of students;

[0017] The emotion recognition module is used to perform feature splicing on the facial expression picture and the physiological signal to obtain a comprehensive feature vector, and recognize the current emotional state of the student based on the comprehensive feature vector and the behavior data;

[0018] The content playing module is used to play the screened audio and video content;

[0019] The information interaction module is used for students to provide feedback on the audio and video content and select preferences.

[0020] Optionally, performing feature splicing on the facial expression image and the physiological signal to obtain a comprehensive feature vector includes:

[0021] The facial expression image and the physiological signal are subjected to feature concatenation using a gradient boosting decision tree model to obtain the comprehensive feature vector.

[0022] Optionally, identifying the student's current emotional state based on the comprehensive feature vector and the behavior data includes:

[0023] Inputting the comprehensive feature vector into an improved ResNet-50 model for recognition, and outputting a high-dimensional feature vector, wherein the improved ResNet-50 model is obtained based on depthwise separable convolution improvement;

[0024] Input the high-dimensional feature vector into a first Bi-LSTM network with an attention mechanism to output a first feature, and input the first feature into a second Bi-LSTM network with an attention mechanism to obtain a sentiment feature sequence;

[0025] Inputting the behavior data into a behavior decision network of Deep Q-Learning and outputting a behavior feature sequence;

[0026] The current emotional state is acquired based on the emotional feature sequence and the behavioral feature sequence.

[0027] Optionally, based on the emotion feature sequence and the behavior feature sequence, acquiring the current emotion state includes:

[0028] Inputting the emotion feature sequence and the behavior feature sequence into a long short-term memory network and outputting a feature matrix;

[0029] The feature of the last time step in the feature matrix is ​​selected and input into the Transformer Encoder model to obtain the feature representation of the current emotional state, and the feature representation of the current emotional state is input into the fully connected layer to obtain the current emotional state.

[0030] Optionally, filtering out audio and video content from the aesthetic education database according to the current emotional state includes:

[0031] The similarity between users is calculated based on the students' rating data and the current emotional state, the recommendation algorithm is optimized based on the similarity, and the audio and video content is screened based on the optimized recommendation algorithm.

[0032] Optionally, the method for calculating the similarity between users based on the student's rating data and the current emotional state is:

[0033]

[0034] Among them, r ui and r vi Represents the ratings of user u and user v on item i, e ui and e vi They represent the current emotional states of users u and v towards item i respectively, and w1 and w2 are the weights of the rating data and the current emotional state respectively.

[0035] Optionally, the administrator client includes: a device management module, a content management module and a data analysis module;

[0036] The device management module is used to check and set the status of the aesthetic education cabin, send device inspection instructions to the student client, and receive status feedback information;

[0037] The content management module is used to add, delete, modify and check the audio and video content in the aesthetic education database;

[0038] The data analysis module is used to perform data statistics and big data analysis on students' emotional states, and to perform statistics and analysis on content interaction data.

[0039] The beneficial effects of the present invention are as follows: the present invention can more accurately understand the emotional state and psychological needs of each student by combining emotion recognition and behavioral psychological correction mechanisms, thereby providing more personalized recommendations for art work content. Such highly personalized works can better meet the unique needs of students, improve the positive emotional intervention effect and student satisfaction. The present invention can intelligently recommend the most suitable art resources according to the emotional state and learning progress of students, avoid the interference of similar resources, and ensure the efficient use of art resources. At the same time, by collecting and analyzing students' feedback and experience data, the art work database and its effects can be better optimized, and the intelligence and resource allocation can be further optimized and improved in iteration. The management interface provided by the present invention can help the system observe the experience state and emotional changes of students in real time, so as to adjust the experience method and content more accurately. This not only reduces the workload of psychological counseling, but also improves the pertinence and efficiency of autonomous experience. The present invention applies the latest artificial intelligence technology to the field of aesthetic education experience, bringing innovation and change to traditional art audio-visual methods. Through the effective improvement of technical means and art works in quality and quantity, the present invention also opens up a path for the new development of aesthetic education.

[0040] To sum up, the present invention, through highly integrated and innovative technical solutions, combines painting, photography and music art creation to provide a new auxiliary means for aesthetic education experience and art therapy, embodies a significant level of personalization, and integrates the effects of artistic experience and the efficiency of aesthetic education resource utilization, which is of great significance to promoting the development of educational technology and improving the quality of education. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1 This is a flow chart of a working method of an audio-visual integrated immersive aesthetic education cabin driven by facial emotion recognition according to an embodiment of the present invention;

[0043] Figure 2 Schematic diagram of a zero-gravity interaction and audio-visual playback device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] like Figure 1 As shown, the present invention discloses an audio-visual integrated immersive aesthetic education cabin driven by facial emotion recognition, including: a student client, an administrator client and a server;

[0047] The student client is used to collect students' facial expression pictures, physiological signals and behavioral data, identify students' current emotional state based on the facial expression pictures, physiological signals and behavioral data, and interact with students; the student client is integrated into the interactive device in the immersive integrated aesthetic education cabin.

[0048] Specifically, the student client includes a data acquisition module, an emotion recognition module, a content playback module and an information interaction module; the data acquisition module is used to collect students' facial expression pictures, physiological signals and behavioral data; the emotion recognition module is used to perform feature splicing on facial expression pictures and physiological signals, obtain comprehensive feature vectors, and identify the students' current emotional state based on the comprehensive feature vectors and behavioral data; the content playback module is used to play the selected audio and video content; the information interaction module is used for students to provide feedback on the audio and video content and select preferences.

[0049] In this embodiment, a high-resolution camera (1080P, 60fps) and a depth sensor (such as Intel RealSense D455) are used to obtain students' behavior and expression data, and the key areas are located based on the YOLOv5 target detection network, and the shooting angle and range of the camera are adjusted in real time to ensure the diversity and accuracy of the collection.

[0050] Further, the emotion recognition module is used to accurately identify the current emotional state of students by analyzing their facial expressions, heart rate and blood pressure physiological data, and behavioral psychological data obtained from behavioral video observations;

[0051] The emotion recognition module includes facial basic expression recognition, micro-expression recognition, and ECG and blood pressure emotion auxiliary recognition based on deep learning. It further captures facial expressions and micro-expressions through a high-definition camera, combines ECG equipment to record ECG signals, and blood pressure monitoring equipment to obtain blood pressure data, and uses a gradient boosting decision tree model to conduct a comprehensive analysis of these data. Features such as heart rate, heart rate variability, facial expression features, and blood pressure changes are extracted, and facial expression features, ECG features, and blood pressure features are spliced ​​into a comprehensive feature vector through a feature splicing method. The gradient boosting decision tree model can handle complex feature relationships and nonlinear data, gradually construct multiple decision trees, and improve classification performance. This embodiment uses a gradient boosting decision tree model to perform feature splicing on facial expression images and physiological signals to obtain a comprehensive feature vector.

[0052] The comprehensive feature vector is converted into basic emotional state. A model combining convolutional neural network (CNN) and recurrent neural network (LSTM) with attention mechanism is adopted, and reinforcement learning is introduced to optimize the decision-making process. First, the improved ResNet-50 is used to replace the convolutional layer in ResNet-50 with depthwise separable convolution for feature recognition, and a 2048-dimensional high-dimensional feature vector is output, which can fully capture the subtle changes in behavior and expression. In order to enhance the robustness of small sample training, group normalization is introduced in the convolutional layer to effectively improve the training stability when the batch size is small. Then, these high-dimensional feature vectors are input into the first layer of Bi-LSTM with multi-head attention mechanism, and the input 2048-dimensional feature vector is encoded into a 1024-dimensional context representation; then, the multi-head attention mechanism is used to focus on the key features in different time segments, with each attention head having a dimension of 64 and a total number of heads of 8, and finally a 1024-dimensional feature representation that integrates temporal information is generated. Next, these features are input into the second-layer Bi-LSTM with a multi-head attention mechanism to further extract the deep dynamic relationships in the time series, and finally output a 512-dimensional emotional feature sequence to describe the changing trend of the emotional state over time, so as to effectively process the feature sequence and capture subtle changes in expression, and generate a more accurate representation of the emotional state. Among them, the multi-head attention mechanism is added to the output of the Bi-LSTM.

[0053] The behavioral psychology correction function collects students' behavioral data through machine vision technology, processes the behavioral data based on the behavior decision network of Deep Q-Learning, and obtains the behavioral feature sequence. Specifically, the features are first reduced to 128 dimensions through a fully connected layer and input into a two-layer fully connected Q network. The first layer has 64 neurons and uses Leaky ReLU as the activation function; the second layer outputs the specific action space dimension for generating behavior adjustment strategies. In order to achieve efficient operation in an edge computing environment, the Deep Q-Learning behavior decision network is weight pruned and quantized, which significantly reduces the computational complexity while ensuring the accuracy of the decision.

[0054] The behavioral feature sequence and the emotional feature sequence are mapped to a 256-dimensional feature space, and the sequence of each sample consists of 30 time steps. The feature representation of each time step is a 256-dimensional vector that contains dynamic information related to behavior and emotion. In order to capture the dependencies and dynamic changes between these time steps, the behavioral feature sequence and the emotional feature sequence are input into the LSTM, which combines a reinforcement learning strategy to optimize the processing of feature sequences and outputs a serialized feature representation that captures the temporal dynamic relationship. When the behavioral feature sequence and the emotional feature sequence are processed, the LSTM outputs a complete feature matrix of shape 30×256, where each row is a 256-dimensional feature representation of a time step. In addition, in order to obtain the current emotional state, the feature of the last time step is selected from these 30 time steps as the main representation of the current state, because it summarizes the dynamic changes of the sequence from the beginning to the end. The features of the last time step output by LSTM are input into the 6-layer Transformer Encoder to complete the output of the feature representation of the current emotional state, and the feature representation of the current emotional state is input into the fully connected layer to obtain the current emotional state. Each layer of the 6-layer Transformer Encoder includes a multi-head attention module (8 heads, each head has a dimension of 64) and a feedforward neural network (two hidden layers, with a dimension of 1024). This structure captures the dependency between features while optimizing the semantic consistency of fused features through global modeling.

[0055] Finally, the entire system is continuously optimized by applying a unified decision-making framework and feedback mechanism in the modules of data collection, emotion recognition, and behavior adjustment. In particular, in the edge computing environment, the accuracy and reliability of data collection are improved by adjusting the camera position and sensor settings; the ResNet-50 optimized with deep separable convolution, the two-layer Bi-LSTM, the multi-head attention mechanism, the Transformer Encoder, and the deep reinforcement learning framework are seamlessly combined to improve the performance of the model while ensuring its efficient operation in the edge computing environment. The content playback module, based on the analysis results of the emotion recognition module, selects audio and video content that can positively guide or intervene in students' emotions from the preset audio-visual integrated aesthetic education database for playback, and the content is presented to students in audio-visual formats such as MP4.

[0056] The information interaction module provides a user interface that allows students to select preferences and adjust recommended settings, and provide feedback on the audio-visual experience. It also accepts device inspection instructions sent by the server and provides feedback on device status.

[0057] Specifically, the server is used to filter out audio and video content from the aesthetic education database according to the current emotional state;

[0058] Furthermore, the server side includes:

[0059] Content recommendation module: Based on the emotion recognition analysis results sent by the student client, a recommendation algorithm is used to select a recommended list of audio and video content that can positively guide or intervene in students' emotions from the audio-visual integrated aesthetic education database and send it back to the corresponding student client;

[0060] The content recommendation module integrates real-time emotion recognition technology and dynamic recommendation algorithms to achieve real-time responsiveness of the recommendation system. Using multiple data sources such as facial expressions, behavioral pattern analysis, and physiological signals, it can capture and analyze the user's current emotional state in real time. Combined with the above current emotional state, the recommendation algorithm dynamically adjusts the content recommendation to ensure that the audio and video content provided is highly relevant to the user's immediate emotional state. This improves the matching degree of the recommended content and enhances the user's interactive experience and satisfaction, providing more personalized and accurate content playback.

[0061] The audio and video content includes art resources of various types and styles, such as music, painting, photography, aerial images and music of students’ hometown scenes familiar to them, etc. It aims to provide different types of content according to user preferences to enrich students’ aesthetic education or emotional intervention experience.

[0062] The information interaction module allows students to set their own preferences, evaluate the relevance and satisfaction of recommended content, collect students' feedback and behavior data, process and analyze these data, and adjust the recommendation algorithm, which includes existing collaborative filtering and content-based algorithms, as well as improved deep learning and hybrid recommendation algorithms. Specifically, based on the students' ratings and emotional states, the similarity weights and content feature weights are recalculated to optimize the parameters and structure of the recommendation model. Advanced technologies such as deep neural networks are used to dynamically adjust the weights and preference layers of the algorithm. The effectiveness of the recommendation system is regularly evaluated, and the performance of different adjustment strategies is compared through A / B testing. New feedback data is continuously collected for iterative optimization, thereby realizing a personalized and intelligent recommendation system to meet the ever-changing needs of students and achieve continuous learning and optimization. Among them, the current emotional state is integrated into the user-item matrix and used together with the rating data to calculate the similarity between users. The user-item matrix is ​​a two-dimensional matrix with rows representing users and columns representing items. Each element r in the matrix ui represents the rating of user u on item i. The current emotional state is obtained by analyzing the user's physiological characteristics such as heart rate, skin galvanic response, facial expression, etc. and behavioral data, and quantified as a value e ui , and integrated into the matrix. When calculating the similarity, the weighted cosine similarity method is used, and the rating data and the current emotional state are given different weights w1 and w2 respectively. The calculation formula for user similarity based on student ratings and current emotional states is as follows:

[0063]

[0064] Among them, r ui and r vi Represents the ratings of user u and user v on item i, e ui and e vi They represent the current emotional states of users u and v towards item i respectively, and w1 and w2 are the weights of the rating data and the current emotional state respectively.

[0065] Using advanced technologies such as deep neural networks, the weights and preference layers of the algorithm are dynamically adjusted. Specifically, a convolutional neural network (CNN) structure is used, which includes multiple convolutional layers and pooling layers, as well as a final fully connected layer. At the same time, the Adam optimizer and cross entropy loss function are selected, which helps to converge quickly and accurately measure the error of category prediction. The network learns through training data and optimizes its weights and biases according to the loss function to reduce errors. After deployment, it is necessary to monitor the performance of the model in real time, and fine-tune or completely update the algorithm based on its performance, and continue to train or adjust the network architecture. This process involves a feedback loop system to ensure that the model can adapt to new data or environmental changes, and maintain its effectiveness and accuracy through evaluation and iteration. Such a system design can not only cope with the initial task requirements, but also adapt to various changes that may occur in the future, ensuring long-term stability and high performance.

[0066] In this way, the content recommendation module can more accurately calculate the similarity between users and recommend items that similar users like. Regularly collecting new ratings and current sentiment states, recalculating similarities and adjusting algorithm parameters ensures that the recommendation system can dynamically adapt to changes in user sentiment and preferences, thereby achieving personalized and intelligent recommendations.

[0067] Student information recording module: interacts with the student client and stores the student's settings, preferences, audio-visual tracks, etc. in the server for easy use and modification by students;

[0068] Data storage module: provides secure storage for audio-visual content and student information. When a student client sends a content request, the requested content is sent to the corresponding student client; when an administrator client sends a content management request, the corresponding operation is performed.

[0069] The server also stores encrypted audio-visual content and student-related information;

[0070] The server also includes an authority verification module, which is used to verify the identity of students and administrators; and a content distribution module, which is used to send content to students or perform management operations of administrators according to requests.

[0071] Administrator client, used by administrators to manage the aesthetic education cabin, aesthetic education database and students' emotional status data.

[0072] The Administrator Client includes:

[0073] The equipment management module provides an interface that allows administrators to manage the art education cabins distributed in a certain space, view and set the status of the art education cabins, send equipment inspection instructions to student clients, and receive feedback on the equipment status of the art education cabins from student clients; the equipment management module allows administrators to manage audio-visual cabins in different locations, retrieve real-time data from the art education cabins, check the equipment status, and monitor and manage the equipment.

[0074] The content management module provides an interface that allows administrators to add, delete, modify, and check video content;

[0075] The data analysis module provides an interface that allows experts with psychological education qualifications to conduct statistics and big data analysis on student information, namely student emotional state data, and to conduct data statistics and analysis on content interaction in order to further promote analysis and strategy adjustment. The data on content interaction refers to student ratings and student preferences.

[0076] like Figure 1 As shown in the figure, the operation process of an audio-visual integrated immersive aesthetic education cabin driven by facial emotion recognition includes:

[0077] Step 1: Setup and Implementation of Emotion Recognition Module:

[0078] 1.1. Install high-resolution cameras in the aesthetic education cabin to capture students’ facial expressions and behaviors.

[0079] 1.2. Install a portable electrocardiograph on the interactive equipment in the aesthetic education cabin to record students’ blood pressure and electrocardiograph data.

[0080] 1.3. Configure emotion recognition software, which can analyze students' facial expressions, blood pressure and electrocardiogram data in real time and identify students' basic emotional states, such as happiness, sadness, anger, etc.

[0081] 1.4. Introduce behavioral analysis algorithms, analyze students’ behavior patterns through video data of student behavior captured by cameras, combined with psychological principles, to assist in the judgment of emotional states.

[0082] 1.5. Integrate the emotion recognition results and behavior analysis results to form a comprehensive assessment of students’ emotional state and provide a basis for the content recommendation module.

[0083] Step 2: Setup and implementation of content recommendation module:

[0084] 2.1. Establish an integrated audio-visual aesthetic education content database, which includes various types of audio-visual aesthetic education resources, and each resource is marked with a suitable emotional state.

[0085] 2.2. Develop a content recommendation algorithm that selects the most matching content from the aesthetic education content database for recommendation based on the students' emotional state provided by the emotion recognition module.

[0086] 2.3. Design an autonomous learning mechanism for the recommendation system, and adjust the recommendation algorithm based on students’ feedback on the recommended content, such as viewing time, interaction frequency, etc., to improve the accuracy and satisfaction of future recommendations.

[0087] Step 3: Setup and implementation of user interaction module, e.g. Figure 2 As shown:

[0088] 3.1. Zero-gravity chairs are equipped in the aesthetic education cabin to provide students with a comfortable emotional adjustment environment. Zero-gravity chairs can adjust postures according to students' body shapes and preferences, creating a human-computer interaction experience in the most relaxed physiological state.

[0089] 3.2. A touch screen is installed next to the zero-gravity chair, and students can interact with the system through the touch screen, such as selecting their favorite content, adjusting the volume and playback speed, and providing evaluation and feedback on the content.

[0090] 3.3. Develop a user interaction interface. The interface design should be simple and intuitive to ensure that students can operate it easily even when they are learning.

[0091] 3.4. Collect students’ feedback and preference information through the user interaction module and feed it back to the content recommendation module to optimize the recommendation strategy.

[0092] Step 4: Setting and implementing the content playback module, such as Figure 2 As shown:

[0093] 4.1. Equipped with high-definition display screen and high-quality surround sound system to ensure the audio-visual effect of aesthetic education content.

[0094] 4.2. Develop content playback control software that supports multiple media formats and can automatically play corresponding audio-visual integrated art content based on the recommendations of the content recommendation module.

[0095] 4.3. Design the playback interface, including basic control functions such as play, pause, fast forward, and fast rewind, and display relevant information of the content, such as the author, historical background, artistic characteristics, etc.

[0096] 4.4. Realize the linkage between the playback module and the user interaction module. For example, students can control the playback progress through the touch screen, and the system adjusts the playback mode according to the students' interaction.

[0097] Step 5: System Integration and Testing:

[0098] 5.1. Integrate the emotion recognition module, content recommendation module, user interaction module and content playback module into a unified system to ensure the efficiency and stability of data transmission and processing between modules.

[0099] 5.2. Carry out system installation and debugging in the aesthetic education cabin to ensure that all equipment and software work together to achieve the expected functions.

[0100] 5.3. Conduct system testing, including functional testing, performance testing and user experience testing, to ensure that the system is stable and reliable, has a user-friendly interface, and can meet students' learning needs.

[0101] 5.4. Optimize the system based on test feedback, adjust parameters, improve interaction design, and enhance user experience.

[0102] Through the above steps, the present invention realizes a highly integrated and personalized aesthetic education learning system, which can provide personalized aesthetic education content recommendations based on students' emotional state and behavioral characteristics, and at the same time improve students' learning experience and effect through a comfortable learning environment and intuitive interactive interface.

[0103] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. An audio-visual integrated immersive aesthetic education cabin driven by facial emotion recognition, characterized in that: include: Student client, administrator client and server; The student client is used to collect facial expression pictures, physiological signals and behavioral data of students, identify the current emotional state of students based on the facial expression pictures, physiological signals and behavioral data, and interact with students; The server side is used to filter out audio and video content from the aesthetic education database according to the current emotional state; The administrator client is used for the administrator to manage the aesthetic education cabin, the aesthetic education database and the students' emotional state data.

2. The audio-visual integrated immersive aesthetic education cabin based on facial emotion recognition drive according to claim 1 is characterized in that: The student client includes a data acquisition module, an emotion recognition module, a content playback module and an information interaction module; The data collection module is used to collect the facial expression pictures, physiological signals and behavioral data of students; The emotion recognition module is used to perform feature splicing on the facial expression picture and the physiological signal to obtain a comprehensive feature vector, and recognize the current emotional state of the student based on the comprehensive feature vector and the behavior data; The content playing module is used to play the screened audio and video content; The information interaction module is used for students to provide feedback on the audio and video content and select preferences.

3. The audio-visual integrated immersive aesthetic education cabin based on facial emotion recognition drive according to claim 2 is characterized in that: Performing feature splicing on the facial expression image and the physiological signal to obtain a comprehensive feature vector includes: The facial expression image and the physiological signal are subjected to feature concatenation using a gradient boosting decision tree model to obtain the comprehensive feature vector.

4. The audio-visual integrated immersive aesthetic education cabin based on facial emotion recognition drive according to claim 2 is characterized in that: Identifying the student's current emotional state based on the comprehensive feature vector and behavior data includes: Inputting the comprehensive feature vector into an improved ResNet-50 model for recognition, and outputting a high-dimensional feature vector, wherein the improved ResNet-50 model is obtained based on depthwise separable convolution improvement; Input the high-dimensional feature vector into a first Bi-LSTM network with an attention mechanism to output a first feature, and input the first feature into a second Bi-LSTM network with an attention mechanism to obtain a sentiment feature sequence; Inputting the behavior data into a behavior decision network of Deep Q-Learning and outputting a behavior feature sequence; The current emotional state is acquired based on the emotional feature sequence and the behavioral feature sequence.

5. The audio-visual integrated immersive aesthetic education cabin based on facial emotion recognition drive according to claim 4 is characterized in that: Based on the emotion feature sequence and the behavior feature sequence, acquiring the current emotion state includes: Inputting the emotion feature sequence and the behavior feature sequence into a long short-term memory network and outputting a feature matrix; The feature of the last time step in the feature matrix is ​​selected and input into the Transformer Encoder model to obtain the feature representation of the current emotional state, and the feature representation of the current emotional state is input into the fully connected layer to obtain the current emotional state.

6. The audio-visual integrated immersive aesthetic education cabin based on facial emotion recognition drive according to claim 3 is characterized in that: The audio and video contents selected from the aesthetic education database according to the current emotional state include: The similarity between users is calculated based on the students' rating data and the current emotional state, the recommendation algorithm is optimized based on the similarity, and the audio and video content is screened based on the optimized recommendation algorithm.

7. The audio-visual integrated immersive aesthetic education cabin based on facial emotion recognition drive according to claim 6 is characterized in that: The method for calculating the similarity between users based on the student's rating data and the current emotional state is: Among them, r ui and r vi Represents the ratings of user u and user v on item i, e ui and e vi They represent the current emotional states of users u and v towards item i respectively, and w1 and w2 are the weights of the rating data and the current emotional state respectively.

8. The audio-visual integrated immersive aesthetic education cabin based on facial emotion recognition drive according to claim 1 is characterized in that: The administrator client includes: a device management module, a content management module and a data analysis module; The device management module is used to check and set the status of the aesthetic education cabin, send device inspection instructions to the student client, and receive status feedback information; The content management module is used to add, delete, modify and check the audio and video content in the aesthetic education database; The data analysis module is used to perform data statistics and big data analysis on students' emotional states, and to perform statistics and analysis on content interaction data.

Images