Classroom group learning emotion compensation method and device based on multiple modes and medium

Abstract

The invention discloses a multi-mode-based classroom group learning emotion compensation method and device and a medium. The method comprises the following steps: acquiring multi-modal data of a classroom teaching process; processing the various multi-modal data by applying different emotion recognition models to obtain emotion recognition results; the emotion recognition result at least comprises a face recognition result, a light recognition result and a voiceprint recognition result; and performing emotion compensation operation on classroom teaching according to the emotion recognition result. According to the method, the multi-modal information of the classroom teaching process is obtained, the corresponding emotion recognition model is used for analyzing the multi-modal information of the classroom to obtain the teaching activity scene of the classroom and the emotional state of the student, and a better teaching atmosphere is provided for the classroom through the preset emotion compensation operation.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion compensation, and specifically designs a method, equipment and medium for classroom group learning emotion compensation based on multimodality. Background Art

[0002] Emotion Recognition is an important application in the field of computer vision, which involves detecting and inferring people's emotional state from multimodal data such as facial expressions, speech, and text.

[0003] Out of consideration for the mental health of teenagers, some educational institutions have introduced advanced technology into daily management to help understand students' mental conditions. At present, these institutions usually follow up on the information they obtain manually in order to provide support to students in need. However, this method may affect the effect due to insufficient response speed, which may not be conducive to timely solving students' emotional problems.

[0004] With the deepening integration of information technology and education, many schools have built smart learning environments. In this process, the application of intelligent lighting systems has become a key factor in improving the teaching experience. Existing intelligent lighting systems mainly adjust classroom lighting based on preset schedules or direct commands from users, and there are no intelligent applications that are deeply integrated with the teaching process. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method, device and medium for classroom group learning emotion compensation based on multimodality.

[0006] The first aspect of the present invention provides a multimodal classroom group learning emotion compensation method, comprising the following steps: Obtain multimodal data of classroom teaching process; Apply different emotion recognition models to various multimodal data to obtain emotion recognition results; the emotion recognition results at least include face recognition results, light recognition results and voiceprint recognition results; According to the emotion recognition results, emotion compensation operations are performed on classroom teaching.

[0007] Further, the multimodal data includes at least video data, illumination data and audio data; The video data is processed by a facial emotion recognition model to obtain facial recognition results that reflect the facial emotions of students in the classroom; The illumination data is processed by a light recognition model to obtain a light recognition result reflecting the illumination and color temperature of the light in the classroom; The audio data is processed by a voiceprint recognition model to obtain a voiceprint recognition result that reflects the content of classroom teaching activities.

[0008] Furthermore, the facial emotion recognition model includes an image segmentation module and an emotion classification module; wherein the image segmentation module is used to perform image segmentation on the input image, and perform feature extraction on each segmented image block to identify the portrait in the image block; the emotion classification module is used to identify the current emotional state of the portrait according to the facial expression of the portrait in the image block; The face recognition result is obtained by the following steps: Performing frame extraction processing on the video data, and using the extracted video frames as input images for a facial emotion recognition model; After the extracted video frames are input into the facial emotion recognition model, each input image is divided into multiple grids through the image segmentation module; Use the image segmentation module to extract features from each segmented image block and identify the portrait in the image block; Recognize the facial expression of the person in the image block by an emotion classification module to obtain the emotional state of the person in the current video frame; The current emotional states of the portraits in multiple video frames are summarized to obtain the real-time emotional sequence of the portraits as the face recognition result.

[0009] Furthermore, the light recognition model includes a color temperature detection module and an illumination detection module; wherein the color temperature detection module is used to obtain the color temperature of the illumination data, and the illumination detection module is used to obtain the illumination of the illumination data; The illumination recognition result is obtained by the following steps: Checking the spectrum of the illumination data in the color temperature checking module to obtain the color temperature of the current classroom environment light; In the illumination inspection module, the illumination data is subjected to photoelectric conversion to obtain the illumination of the current classroom environment light; The color temperature and illumination of the current classroom ambient light are summarized to obtain the real-time ambient light index as the light recognition result.

[0010] Furthermore, the voiceprint recognition model includes a speaker separation module and a voiceprint recognition module; the speaker separation module is used to identify the voiceprint data of different speakers in the audio data; the voiceprint recognition module is used to identify the specific speaker of different voiceprint data according to a preset class voiceprint library; The voiceprint recognition result is obtained by the following steps: Input the audio data into the speaker separation module to obtain multiple groups of voiceprint data; The multiple groups of voiceprint data are input into a voiceprint recognition module, and the speaking object of each group of voiceprint data is identified as a voiceprint recognition result.

[0011] Furthermore, the speaker separation module specifically identifies the voiceprint data of different speakers in the audio data through the following steps: Converting the audio data into an STFT power spectrum by short-time Fourier transform; Map the STFT power spectrum to the Mel frequency scale to obtain the Mel spectrum; Using a linear time series model to extract voiceprint features from the Mel spectrum to obtain a multi-frame feature matrix; A clustering algorithm is used to perform feature clustering on the voiceprint features in the multi-frame feature matrix to obtain multiple feature clusters, and the voiceprint features under each feature cluster respectively represent the voiceprint data of a speaker.

[0012] Furthermore, the emotion compensation operation is performed on classroom teaching according to the emotion recognition result, which specifically includes the following steps: Determine the current classroom teaching scene through the voiceprint recognition result; Determining the emotional state of the current class through the face recognition results; Apply emotional compensation strategies according to the current classroom teaching scenario and emotional state, and perform emotional compensation operations under the corresponding emotional compensation strategies.

[0013] Furthermore, the emotion compensation operation specifically includes adjusting the classroom lighting atmosphere and adjusting the classroom speaker music; The classroom lighting atmosphere is adjusted according to the light recognition result by using the color temperature adjustment algorithm and the light intensity adjustment algorithm set in the corresponding emotion compensation strategy; The adjustment of the classroom loudspeaker music is specifically performed through the music playing logic set in the corresponding emotion compensation strategy.

[0014] A second aspect of the present invention discloses an electronic device, comprising a processor and a memory; The memory is used to store programs; The processor executes the program to implement the multimodal-based classroom group learning emotion compensation method.

[0015] The third aspect of the present invention discloses a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the multimodal-based classroom group learning emotion compensation method.

[0016] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0017] The embodiments of the present invention have the following beneficial effects: the method, device and medium for classroom group learning emotion compensation based on multimodality obtain multimodal information of the classroom teaching process, use the corresponding emotion recognition model to analyze the classroom teaching activity scene and the emotional state of the students from the multimodal information of the classroom, and provide a better teaching atmosphere for the classroom through the preset emotion compensation operation. The embodiments of the present invention are widely used in classroom teaching in on-campus and off-campus institutions to create a good teaching environment for teachers and students and improve teaching effects.

[0018] Additional aspects and advantages of the present invention will be given in part in the following description and, in part, will be obvious from the following description or learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 It is a main implementation flow chart of a multimodal classroom group learning emotion compensation method of the present invention.

[0021] Figure 2 It is a flow chart of video data processing in the method of the present invention.

[0022] Figure 3 It is a flow chart of illumination data processing in the method of the present invention.

[0023] Figure 4 It is a flow chart of audio data processing in the method of the present invention.

[0024] Figure 5 It is a detailed implementation flow chart of the method of the present invention.

[0025] Figure 6 It is a structural diagram of an electronic device of the present invention.

[0026] Figure 7 It is a structural diagram of a computer-readable storage medium of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0028] The present invention aims to solve at least one of the related problems existing in the prior art. To this end, the present invention provides a multimodal classroom group learning emotion compensation method, system, device and storage medium.

[0029] like Figure 1 As shown, the first embodiment of the present invention provides a multimodal classroom group learning emotion compensation method, comprising the following steps: S1. Obtain multimodal data of classroom teaching process; S2. Apply different emotion recognition models to process various multimodal data to obtain emotion recognition results; S3. Perform emotion compensation operations on classroom teaching based on the results of emotion recognition.

[0030] The implementation process of each step of the present invention is described in detail below: S1. Obtain multimodal data of classroom teaching process.

[0031] In the embodiment of the present invention, multimodal data refers to a variety of heterogeneous data generated in the classroom teaching process. By collecting these multimodal data to predict the classroom teaching atmosphere and performing targeted emotional compensation operations, a better teaching atmosphere can be provided for classroom teaching.

[0032] As a preferred embodiment, the multimodal data obtained in the embodiment of the present invention includes video data, lighting data and audio data. Among them, the video data is obtained by a camera installed in the classroom, the lighting data is obtained by a lighting sensor installed in the classroom, and the audio data is obtained by a microphone installed in the classroom. In another part of the embodiments, under the premise of setting the corresponding emotion compensation operation, the multimodal data obtained can also include teaching time, teaching content, etc., to perform more comprehensive classroom teaching emotion compensation.

[0033] S2. Apply different emotion recognition models to process various multimodal data to obtain emotion recognition results.

[0034] In a preferred embodiment, the emotion recognition results obtained by applying different emotion recognition models to various multimodal data include at least face recognition results, light recognition results and voiceprint recognition results. Among them, the video data is processed by the face emotion recognition model to obtain face recognition results reflecting the facial emotions of students in the classroom; the light data is processed by the light recognition model to obtain light recognition results reflecting the illumination and color temperature of the lights in the classroom; the audio data is processed by the voiceprint recognition model to obtain voiceprint recognition results reflecting the content of classroom teaching activities.

[0035] The following specifically describes the processing flow of the facial emotion recognition model, light recognition model and voiceprint recognition model in the embodiments of the present invention.

[0036] Facial emotion recognition model: Figure 2 As shown, the facial emotion recognition model of the embodiment of the present invention includes an image segmentation module and an emotion classification module; wherein the image segmentation module is used to perform image segmentation on the input image, and perform feature extraction on each segmented image block to identify the portrait in the image block; the emotion classification module is used to identify the current emotional state of the portrait according to the facial expression of the portrait in the image block.

[0037] The face recognition results are obtained through the following steps: Perform frame extraction on the video data, and use the extracted video frames as input images for the facial emotion recognition model; After the extracted video frames are input into the facial emotion recognition model, each input image is divided into multiple grids through the image segmentation module; Use the image segmentation module to extract features from each segmented image block and identify the portrait in the image block; The facial expression of the person in the image block is recognized by the emotion classification module to obtain the emotional state of the person in the current video frame; The current emotional states of the portraits in multiple video frames are summarized to obtain the real-time emotional sequence of the portraits as the face recognition result.

[0038] As a specific embodiment, the image segmentation module can be implemented using the YOLOv11 model. After the input image is divided into S×S grids, each grid is responsible for detecting objects within its area. Image features are extracted through a deep neural network to generate a feature map. Each grid unit predicts a fixed number of bounding boxes, and calculates the confidence of each bounding box, and finally retains the bounding box with the highest confidence as the face recognition result.

[0039] On the other hand, the emotion classification module can be realized by Convolutional Neural Networks (CNN). By collecting student images with various facial expressions and annotating their emotional states in advance, using the annotated student images to establish a data set to train the convolutional neural network, and optimizing the convolutional neural network through the cross entropy damage function, we finally get a convolutional neural network that can effectively extract emotional information from video frames as an emotion classification module. Finally, we summarize the current emotional states of the portraits in multiple video frames to get the real-time emotional sequence of the portraits.

[0040] Light recognition model: Figure 3 As shown, the light recognition model of the embodiment of the present invention includes a color temperature checking module and an illumination checking module; wherein the color temperature checking module is used to obtain the color temperature of the illumination data, and the illumination checking module is used to obtain the illumination of the illumination data.

[0041] The illumination recognition result is obtained through the following steps: Check the spectrum of the lighting data in the color temperature check module to obtain the color temperature of the current classroom ambient light; In the illumination inspection module, the illumination data is converted into photoelectricity to obtain the illumination of the current classroom environment; The color temperature and illumination of the current classroom ambient light are summarized to obtain the real-time ambient light index as the light recognition result.

[0042] In the embodiment of the present invention, the color temperature inspection module is implemented by a spectrometer. The spectrometer collects the spectrum of the illumination data through a photosensitive element, and then uses a preset spectrum algorithm to calculate the color temperature of the classroom light. The illumination inspection module is implemented by an illuminometer, which performs photoelectric conversion on the illumination data in the illuminometer, excites electrons to form current, and obtains the illumination of the classroom light by measuring the current value.

[0043] Voiceprint recognition model: Figure 4 As shown, the voiceprint recognition model includes a speaker separation module and a voiceprint recognition module; the speaker separation module is used to identify the voiceprint data of different speakers in the audio data; the voiceprint recognition module is used to identify the specific speaking objects of different voiceprint data according to the preset class voiceprint library.

[0044] The voiceprint recognition result is obtained through the following steps: Input the audio data into the speaker separation module to obtain multiple groups of voiceprint data; Multiple groups of voiceprint data are input into the voiceprint recognition module, and the speaker of each group of voiceprint data is identified as the voiceprint recognition result.

[0045] As a preferred embodiment, the speaker separation module specifically identifies the voiceprint data of different speakers in the audio data through the following steps: Convert audio data into STFT power spectrum through short-time Fourier transform; Map the STFT power spectrum to the Mel frequency scale to obtain the Mel spectrum; Use the linear time series model to extract voiceprint features from the Mel spectrum and obtain a multi-frame feature matrix; A clustering algorithm is used to perform feature clustering on the voiceprint features in the multi-frame feature matrix to obtain multiple feature clusters. The voiceprint features under each feature cluster represent the voiceprint data of a speaker.

[0046] Specifically, the audio signal is processed using a short-time Fourier transform (STFT) to convert the time domain signal into a time-frequency domain feature. As a preferred embodiment, the audio signal Under STFT transformation, we get: ; in, is the original audio signal, Is a window function, used to split the audio signal, is the time index, is the frequency index, is the size of the window.

[0047] Map the power spectrum after STFT transformation to the Mel frequency scale to obtain the Mel spectrum: ; in, is the STFT power spectrum, It is a Mel filter, which is used to adjust the frequency distribution. This type of time-frequency image is used as the input of the deep neural network for subsequent training.

[0048] These time-frequency features are then further processed using a linear time series model with selective state space (Mamba-2) to learn more complex temporal dependencies and local features, thereby improving the model's ability to recognize sound signals.

[0049] The extracted features are processed by the Mamba-2 model to generate high-dimensional feature representations. The Mamba-2 network is used to capture the temporal dependencies in speech signals, and the output is the corresponding high-order feature representation. The Mamba-2 network includes an input layer, a Mamba-2 layer, a fully connected layer, and an output layer. The Mamba-2 network can effectively map the Mel-spectrogram features to high-dimensional feature representations and enhance the distinguishability of the features. After mapping, a matrix containing multi-frame features is finally obtained, with a shape of T×D, where T is the number of time frames and D is the feature dimension.

[0050] The obtained features are clustered. The embodiment of the present invention uses the K-means algorithm to determine the cluster center and assigns the feature vectors to different clusters, each cluster representing a speaker.

[0051] The speaker separation module is trained using Deep Clustering Loss. First, define a label matrix ,in Indicates time frame Belongs to the speaker ,and tc =0 means not belonging to the speaker . Using this label matrix, we can calculate the similarity matrix of the real label: ; The elements of this matrix Indicates time frame and Do they belong to the same cluster? The output of the network is an embedding matrix , the embedded similarity matrix can be calculated through the dot product: ; The elements of this matrix Represents a time frame in the embedding space and The final loss function is the Frobenius norm difference between these two similarity matrices: ; in, represents the Frobenius norm, which calculates the square difference of the elements of the two matrices. The goal of this loss function is to minimize the label similarity matrix and embedding similarity matrix The difference between them is used to learn the correct embedding representation. The Adam optimization algorithm is selected in the training process to update the network parameters through back propagation and tune the hyperparameters. The training is iterated until the loss function converges or the set stop condition is reached.

[0052] The class voiceprint library of the embodiment of the present invention stores the audio information of each class member. It contains the binary data of the audio sample and the class member label. The probability of each cluster belonging to each speaker is determined by the following formula: ; in, and Respectively with category The relevant weight and bias parameters, is the total number of categories (i.e., the number of all speakers), are input features.

[0053] Use forward propagation to input training samples, calculate the output through the neural network and calculate the loss value, and use the cross entropy loss function to measure the difference between the probability results predicted by the model and the actual label. The loss function formula is: ; For each sample , calculate the true label and the probability distribution predicted by the model The difference between The losses of the samples are summed up to get the total loss value of the entire batch.

[0054] Finally, the obtained voiceprint data is matched with the class voiceprint library to confirm the identity information of the current speaker.

[0055] S3. Perform emotion compensation operations on classroom teaching based on the results of emotion recognition.

[0056] In the embodiment of the present invention, the voiceprint recognition result is used to determine the current classroom teaching scene. Exemplarily, the teaching scene can be determined by the following steps: If the current voice segment has no speaker information, it is determined that the current classroom teaching activity content is self-study (examination); If the speaker of the current voice segment is only the teacher, then the current classroom teaching activity content is determined to be teaching; If the speaker of the current voice segment is multiple people including the teacher, then the current classroom teaching activity content is determined to be question-and-answer; If the speakers of the current voice clip are multiple people excluding the teacher, the current classroom teaching activity content is determined to be discussion, and the number of people currently participating in the discussion is recorded.

[0057] In the embodiment of the present invention, the face recognition result determines the emotional state of the current class, and applies the emotion compensation strategy according to the teaching scene and emotional state of the current class, and performs the emotion compensation operation under the corresponding emotion compensation strategy.

[0058] In the embodiment of the present invention, the emotion compensation operation includes adjusting the classroom lighting atmosphere and adjusting the classroom loudspeaker music. The classroom lighting atmosphere is adjusted according to the light recognition result by the color temperature adjustment algorithm and the light intensity adjustment algorithm set in the corresponding emotion compensation strategy; the classroom loudspeaker music is adjusted by the music playing logic set in the corresponding emotion compensation strategy.

[0059] As a specific embodiment, the color temperature of the classroom lighting atmosphere is adjusted by the following algorithm: ; in: The value of can be 1, 2, 3, 4, 5, representing teaching, discussion, self-study, question and answer, and break scenes respectively; Indicates in the scene situation; It indicates the situation when the number of students with emotions such as sleepiness reaches a threshold; It indicates what happens when the number of students with emotions such as frustration reaches a threshold; Indicates the situation when the proportion of participants in a discussion scenario is lower than the threshold; The color temperature is always kept at Within the range, Take 3000K, Take 5000K.

[0060] As a specific embodiment, the light intensity adjustment of the classroom lighting atmosphere is performed by the following algorithm: ; in: Indicates the current light intensity; is the adjustment factor.

[0061] As a specific embodiment, the music playing logic includes the following judgment logic: ; in: It means starting the classroom speakers to play soothing music; It means starting the classroom speakers to play light electronic pure music.

[0062] The overall implementation process of the embodiment of the present invention is as follows Figure 5 As shown. Based on the classroom teaching activity scene, combined with the students' emotional state, natural lighting conditions and relevant principles of lighting psychology, the preset emotion compensation algorithm is used to intelligently adjust the lighting atmosphere and the state of the audio equipment in the classroom. The intelligent adjustment and compensation of the classroom atmosphere and students' emotions are achieved through changes in lighting conditions and audio content, thereby improving the teaching or activity effects in the classroom.

[0063] As a specific embodiment, the application effects of the embodiments of the present invention in different classroom scenarios are given below: In the teaching scenario, the facial emotion recognition model obtains the current emotion sequence of all students in the class. After statistical analysis of the students' emotional states and combined with the color temperature adjustment algorithm of the classroom lighting atmosphere, it can be seen that the light color temperature (+100K) is increased in the teaching scenario. Increasing the light color temperature can stimulate the brain, keep awake, and help students better focus on the course content; at the same time, increasing the lighting intensity (+100Lux) makes the classroom environment brighter and visually clearer, which helps students see the blackboard, books or screen content clearly, reduces eye fatigue, and makes it easier for students to focus on reading, writing and thinking, thereby improving learning efficiency; when the proportion of students with high or medium pleasure (Valence) and low arousal (Arousal) emotions such as sleepiness reaches 10%, the light color temperature is increased (+200K) to improve students' concentration; when the proportion of students with high or medium arousal and low pleasure emotions such as depression reaches 5%, the light color temperature is reduced (-200K) to alleviate students' negative emotions. In addition, combined with the real-time natural light intensity data obtained by the light sensor, the classroom lights will automatically dim or fill in the light according to the light intensity adjustment algorithm, thereby improving students' learning effects.

[0064] In the discussion scenario, the facial emotion recognition model obtains the current emotional sequence of all students in the class. Through statistical analysis of students' emotional states, when the proportion of students with high or medium valence and low arousal (Arousal) emotions such as sleepiness reaches 10%, the light color temperature is increased (+200K) to improve students' concentration; when the proportion of students with high or medium arousal and low valence emotions such as depression reaches 5% or the current number of participants in the discussion is less than 50%, the light color temperature is reduced (-200K) to relieve students' negative emotions, create a relaxing and warm environment, reduce tension, and encourage interaction and discussion among students. At the same time, according to the logic of emotional compensation music playback, the classroom speakers will automatically start playing light electronic pure music in the discussion session. The relaxed rhythm and melody help create an active classroom atmosphere, stimulate students' creative thinking, and make students more willing to participate in the discussion. In addition, combined with the real-time natural light intensity data obtained by the light sensor, the classroom lights will adjust the algorithm according to the light intensity. When the current number of participants in the discussion is less than 50%, the lighting intensity will be increased (+100Lux) to maintain students' attention. Strong light can activate students' thinking, keep them focused, and promote more communication and cooperation.

[0065] In the self-study and question-answering scenarios, the facial emotion recognition model obtains the current emotion sequence of all students in the class. After statistical analysis of the students' emotional states and combining the color temperature adjustment algorithm of the classroom lighting atmosphere, it can be seen that when the proportion of students with high or medium valence and low arousal (Arousal) emotions such as sleepiness reaches 10%, the light color temperature is increased (+200K) to improve students' concentration; when the proportion of students with high or medium arousal and low arousal emotions such as depression reaches 5%, the light color temperature is reduced (-200K) to alleviate students' negative emotions. In addition, combined with the real-time natural light intensity data obtained by the light sensor, the classroom lights will automatically dim or fill in the light appropriately according to the light intensity adjustment algorithm, thereby improving students' learning effects.

[0066] In the break scene, the facial emotion recognition model obtains the current emotion sequence of all students in the class. After statistical analysis of the students' emotional state and combined with the color temperature adjustment algorithm of the classroom lighting atmosphere, it can be seen that the color temperature of the light is lowered (-200K) during the break to create a relaxed and comfortable classroom environment, making students feel warm and safe. At the same time, the speakers in the classroom play soothing light music to create a warm and pleasant classroom environment, which helps to improve emotions, relieve anxiety or fatigue, and enhance pleasure. At the same time, according to the logic of emotional compensation music playback, when the proportion of students with high or medium pleasure (Valence) and low arousal (Arousal) emotions such as sleepiness reaches 30%, the classroom speakers will automatically start playing light music with a soothing rhythm to help students restore their energy. In addition, combined with the real-time natural light intensity data obtained by the light sensor, the smart light strip will automatically perform appropriate dimming or fill-in light according to the light intensity adjustment algorithm, thereby creating a positive and energetic learning environment.

[0067] Figure 6 : is a schematic diagram of the structure of the electronic device proposed in the second embodiment of the present invention. The memory of this embodiment stores program instructions for implementing the classroom group learning emotion compensation method of any of the above embodiments. The processor is used to execute the program instructions stored in the memory to perform classroom group learning emotion compensation. Among them, the processor can also be called CPU (Central Processing Unit). The processor may be an integrated circuit chip with signal processing capabilities. The processor can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0068] The contents of the method in the first embodiment of the present invention are all applicable to the electronic device embodiment. The functions specifically implemented by the electronic device embodiment are the same as those of the above method embodiment, and the beneficial effects achieved are also the same as those achieved by the above method.

[0069] Figure 7 It is a schematic diagram of the structure of the computer-readable storage medium of the third embodiment of the present invention. The computer-readable storage medium of the fourth embodiment of the present invention stores program instructions that can implement the above-mentioned classroom group learning emotion compensation method, wherein the program instructions can be stored in the above-mentioned storage medium in the form of a software product, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present invention. The aforementioned computer-readable storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes, or terminal devices such as computers, servers, mobile phones, tablets, etc.

[0070] The contents of the method in the first embodiment of the present invention are all applicable to the computer-readable storage medium embodiment. The functions specifically implemented by the computer-readable storage medium embodiment are the same as those of the above method embodiment, and the beneficial effects achieved are also the same as those achieved by the above method.

[0071] This embodiment also provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the above-mentioned related steps to implement the classroom group learning emotion compensation method provided by the above embodiment.

[0072] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0073] Those skilled in the art will appreciate that the modules in the devices in the embodiments of the present invention may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments of the present invention may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the corresponding claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the corresponding claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0074] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0075] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0076] In addition, each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. In particular, for embodiments such as devices and equipment, since they are basically similar to method embodiments, the relevant parts can refer to the partial description of the method embodiments. The embodiments of the devices and equipment described above are merely schematic, wherein the modules, units, etc. described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed in multiple places, such as nodes of a system network. Specifically, some or all of the modules and units may be selected according to actual needs to achieve the purpose of the above-mentioned embodiment scheme. Those skilled in the art can understand and implement it without paying creative labor.

[0077] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0078] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0079] In addition, the terms "first", "second", etc. used in the embodiments of the present invention are only used for descriptive purposes and should not be understood as indicating or implying relative importance, or implicitly indicating the number of technical features indicated in the present embodiment. Therefore, the features defined by the terms "first", "second", etc. in the embodiments of the present invention can explicitly or implicitly indicate that the embodiment includes at least one of the features. In the description of the present invention, the word "multiple" means at least two or two or more, such as two, three, four, etc., unless otherwise clearly and specifically defined in the embodiments.

[0080] In the embodiments of the present invention, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element. In addition, components, features, and elements with the same name in different embodiments of the present invention may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context in the specific embodiment.

[0081] Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and cannot be construed as limitations of the present invention, and those of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention. Those skilled in the art will readily come to think of other embodiments of the present invention after considering the specification and practicing the present invention. This application is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art that are not disclosed in the present invention. The specification and embodiments are intended to be exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

Claims

1. A multimodal classroom group learning emotion compensation method, characterized in that: The following steps are involved: Obtain multimodal data of classroom teaching process; Different emotion recognition models are applied to various multimodal data to obtain emotion recognition results; The emotion recognition result includes at least a face recognition result, a light recognition result and a voiceprint recognition result; According to the emotion recognition results, emotion compensation operations are performed on classroom teaching.

2. According to the multimodal classroom group learning emotion compensation method described in claim 1, it is characterized in that: The multimodal data includes at least video data, lighting data and audio data; The video data is processed by a facial emotion recognition model to obtain facial recognition results that reflect the facial emotions of students in the classroom; The illumination data is processed by a light recognition model to obtain a light recognition result reflecting the illumination and color temperature of the light in the classroom; The audio data is processed by a voiceprint recognition model to obtain a voiceprint recognition result that reflects the content of classroom teaching activities.

3. According to the multimodal classroom group learning emotion compensation method of claim 2, it is characterized in that: The facial emotion recognition model includes an image segmentation module and an emotion classification module; wherein the image segmentation module is used to segment the input image, and extract features from each segmented image block to identify the portrait in the image block; the emotion classification module is used to identify the current emotional state of the portrait according to the facial expression of the portrait in the image block; The face recognition result is obtained by the following steps: Performing frame extraction processing on the video data, and using the extracted video frames as input images for a facial emotion recognition model; After the extracted video frames are input into the facial emotion recognition model, each input image is divided into multiple grids through the image segmentation module; Use the image segmentation module to extract features from each segmented image block and identify the portrait in the image block; Recognize the facial expression of the person in the image block by an emotion classification module to obtain the emotional state of the person in the current video frame; The current emotional states of the portraits in multiple video frames are summarized to obtain the real-time emotional sequence of the portraits as the face recognition result.

4. According to the multimodal classroom group learning emotion compensation method of claim 2, it is characterized in that: The light recognition model includes a color temperature detection module and an illumination detection module; wherein the color temperature detection module is used to obtain the color temperature of the illumination data, and the illumination detection module is used to obtain the illumination of the illumination data; The illumination recognition result is obtained by the following steps: Checking the spectrum of the illumination data in the color temperature checking module to obtain the color temperature of the current classroom environment light; In the illumination inspection module, the illumination data is subjected to photoelectric conversion to obtain the illumination of the current classroom environment light; The color temperature and illumination of the current classroom ambient light are summarized to obtain the real-time ambient light index as the light recognition result.

5. The method for emotional compensation in classroom group learning based on multimodality according to claim 2 is characterized in that: The voiceprint recognition model includes a speaker separation module and a voiceprint recognition module; the speaker separation module is used to identify the voiceprint data of different speakers in the audio data; the voiceprint recognition module is used to identify the specific speaker of different voiceprint data according to the preset class voiceprint library; The voiceprint recognition result is obtained by the following steps: Input the audio data into the speaker separation module to obtain multiple groups of voiceprint data; The multiple groups of voiceprint data are input into a voiceprint recognition module, and the speaking object of each group of voiceprint data is identified as a voiceprint recognition result.

6. The method for emotional compensation in classroom group learning based on multimodality according to claim 5 is characterized in that: The speaker separation module specifically identifies the voiceprint data of different speakers in the audio data by the following steps: Converting the audio data into an STFT power spectrum by short-time Fourier transform; Map the STFT power spectrum to the Mel frequency scale to obtain the Mel spectrum; Using a linear time series model to extract voiceprint features from the Mel spectrum to obtain a multi-frame feature matrix; A clustering algorithm is used to perform feature clustering on the voiceprint features in the multi-frame feature matrix to obtain multiple feature clusters, and the voiceprint features under each feature cluster respectively represent the voiceprint data of a speaker.

7. The method for emotional compensation in classroom group learning based on multimodality according to claim 1 is characterized in that: The emotion compensation operation for classroom teaching according to the emotion recognition result specifically includes the following steps: Determine the current classroom teaching scene through the voiceprint recognition result; Determining the emotional state of the current class through the face recognition results; Apply emotional compensation strategies according to the current classroom teaching scenario and emotional state, and perform emotional compensation operations under the corresponding emotional compensation strategies.

8. The method for emotional compensation in classroom group learning based on multimodality according to claim 7 is characterized in that: The emotional compensation operation specifically includes adjusting the classroom lighting atmosphere and adjusting the classroom speaker music; The classroom lighting atmosphere is adjusted according to the light recognition result by using the color temperature adjustment algorithm and the light intensity adjustment algorithm set in the corresponding emotion compensation strategy; The adjustment of the classroom loudspeaker music is specifically performed through the music playing logic set in the corresponding emotion compensation strategy.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement a multimodal classroom group learning emotion compensation method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement a multimodal classroom group learning emotion compensation method as described in any one of claims 1-8.

Images