Emotion recognition method and system for teaching assistant robot

Abstract

The invention discloses an emotion recognition method and system for a teaching assistant robot, and relates to the technical field of emotion recognition, and the method comprises the steps: constraining the frequency of data collection, and summarizing the obtained data to generate a multi-modal emotion data set; labeling the data in the multi-modal emotion data set, and preprocessing the labeled multi-modal emotion data; the multi-modal emotion data collected in real time is used as input, the trained emotion recognition classification model outputs a corresponding recognition result, and a corresponding response strategy is matched for the teaching assistant robot according to the recognition result; and after the teaching assisting robot makes a response, an improvement coefficient is constructed according to feedback data changes in the observation period, and if the improvement coefficient does not exceed an improvement threshold value, a response strategy is optimized and adjusted. Response can be made quickly, and the teaching quality is improved; the response strategy is predicted by constructing a teaching feedback prediction model, and the effectiveness of the response strategy can also be ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion recognition, and in particular to an emotion recognition method and system for a teaching assistant robot. Background Art

[0002] The teaching assistant robot is an educational auxiliary tool that integrates artificial intelligence technology. It can not only automatically answer questions students encounter in the learning process and provide personalized learning resources and path planning, but also work with teachers to improve teaching efficiency and interactivity, creating a more intelligent, flexible and personalized learning environment for students. Through advanced technologies such as natural language processing and machine learning, the teaching assistant robot can deeply understand students' learning needs and provide precise teaching support.

[0003] In the Chinese invention patent with application publication number CN113160629A, an educational robot for human-machine collaborative learning with emotion recognition function is disclosed, including a main body, a head connected to the top of the main body, a ring-shaped charging slot on the back, an auxiliary bracelet installed in the ring-shaped charging slot, a chip installed inside the head, a learning system carried in the chip, the learning system includes a central processing module, a learning database, a login subsystem, an emotion recognition subsystem, an audio-visual output subsystem, a monitoring subsystem, and a communication subsystem, and the learning system communicates with the auxiliary bracelet through the communication subsystem. The invention is flexible in use, and when learning, it coordinates the students' emotions, vision and hearing to improve the students' learning efficiency and quality.

[0004] In the Chinese invention patent application publication number CN110119715A, a companion robot and an emotion recognition method are provided, wherein a companion robot includes an image information and voice information acquisition device, a controller and a speaker, the image information and voice information acquisition device and the speaker are electrically connected to the controller, the image information and voice information acquisition device is used to collect image information and voice information of the accompanied person, and send the image information and voice information of the accompanied person to the controller, the controller includes an emotion recognition module and a chat control unit, the emotion recognition module is used to judge the emotional pattern of the accompanied person according to the image information and voice information of the accompanied person, and send the emotional pattern of the accompanied person to the chat control unit, and the chat control unit outputs voice information through the speaker according to the emotional pattern of the accompanied person.

[0005] Combined with the above application and the contents of the prior art: When educational robots are used in teaching activities, students' emotional changes can provide feedback on the current teaching status and teaching results. When teachers identify such emotional changes, they can respond and provide feedback on the current teaching process, and adjust and improve the current teaching results.

[0006] Especially when students in the teaching area have low interactive participation and low interaction frequency, if the teaching and response strategies of the teaching assistant robots cannot be adjusted in time, the current teaching effect may decline rapidly; but when existing teaching assistant robots identify students' emotions and respond accordingly, they usually need to send the emotion recognition results to the administrators, who will give adjusted response strategies. This can easily lead to the teaching assistant robots responding slowly after completing emotion recognition, and failing to quickly adjust the current poor teaching feedback.

[0007] To this end, the present invention provides an emotion recognition method and system for a teaching assistant robot. Summary of the invention

[0008] 1. Technical issues to be resolved In view of the deficiencies of the prior art, the present invention provides an emotion recognition method and system for a teaching assistant robot, which labels the data in a multimodal emotion data set, preprocesses the labeled multimodal emotion data, and obtains the preprocessed data; the multimodal emotion data collected in real time is used as input, and the trained emotion recognition classification model outputs the corresponding recognition result, and the corresponding response strategy is matched for the teaching assistant robot according to the recognition result; after the teaching assistant robot responds, an improvement coefficient is constructed according to the change of feedback data within the observation period, and if the improvement coefficient does not exceed the improvement threshold, the response strategy is optimized and adjusted. By constructing a teaching feedback prediction model to predict the response strategy, the effectiveness of the response strategy can also be ensured, thereby solving the technical problems in the background technology.

[0009] (II) Technical solution To achieve the above objectives, the present invention is implemented through the following technical solutions: An emotion recognition method for a teaching assistant robot, comprising collecting feedback data from students during class teaching, and generating a feedback coefficient in the current teaching activity process from the teaching feedback data , if the feedback coefficient If the value is lower than the feedback threshold, a data collection instruction is issued to the outside; The path planning algorithm plans a data collection path in the teaching area, and multimodal emotion data is collected in the teaching area in turn according to the data collection path, and the frequency of data collection is constrained, and the acquired data is aggregated to generate a multimodal emotion data set; Label the data in the multimodal emotion data set, and after similarity analysis, generate the labeling accuracy based on the similarity analysis data. If the marking accuracy If the value is higher than the precision threshold, the labeled multimodal emotion data is preprocessed to obtain preprocessed data; Train and acquire the emotion recognition classification model. After testing it, verify its recognition effect with the acquired test data. If the recognition effect meets expectations, the multimodal emotion data collected in real time is used as input, and the trained emotion recognition classification model outputs the corresponding recognition result. According to the recognition result, the corresponding response strategy is matched for the teaching assistant robot. After the teaching assistant robot responds, the improvement coefficient is constructed based on the feedback data changes during the observation period. , if the improvement coefficient Without exceeding the improvement threshold, the response strategy is optimized and adjusted, and the optimized response strategy is verified by the teaching feedback prediction model to obtain the improvement coefficient again. , if the improvement coefficient If it is still lower than expected, the response strategy will be re-optimized.

[0010] Furthermore, during the monitoring period, feedback data from students during class teaching is collected, and the obtained feedback data is aggregated to generate a teaching feedback data set; feedback coefficients are generated from the teaching feedback data set. ,in, On participation and interaction frequency Perform linear normalization and map the corresponding data values ​​to the interval In the following way: ; Weight coefficient: , and ; For the i The degree of participation of students in teaching activities, is the qualified standard value of participation, is the mean of participation; For the i The frequency of interaction between students in teaching activities, is the qualified standard value of interaction frequency, is the mean interaction frequency.

[0011] Further, after receiving the data collection instruction, the student distribution area is divided into several sub-areas, and after constructing an electronic map of the student distribution area, each sub-area is marked on the electronic map; The feedback coefficient and position information in each sub-area are obtained, and the path planning algorithm is used to plan the path for each sub-area to obtain the corresponding data collection path.

[0012] Furthermore, after determining the data collection order, the data collection types are limited. When collecting multimodal emotion data in each sub-area, the frequency of data collection is constrained. Data is collected in the sub-area at the data collection nodes that meet the constraint conditions. The constraint method is as follows: ; in, n is the number of nodes collecting multimodal emotion data, It is i The collection node to the j The time interval of each collection node, is the average value of the time interval, is the feedback coefficient.

[0013] Furthermore, after obtaining the data in the multimodal emotion data set, the selected experts annotate the data in detail, and the annotated data is used as the first annotated set; multiple other annotators annotate the same data, obtain the annotated data and use it as the second annotated set; A similarity analysis is performed on the two annotation results in the first annotation set and the second annotation set, and a corresponding similarity data set is generated after obtaining similarity data.

[0014] Furthermore, under dimensionless conditions, the annotation accuracy is generated from the similarity data in the similarity data set. , as follows: ; in, n is the number of annotators, It is i The similarity of data annotation between annotators and experts, is the average similarity value, is the qualified target value of similarity; weight coefficient: , ,and .

[0015] Furthermore, after obtaining the preprocessed data, some data is selected from the preprocessed data as sample data, and the sample data is divided into a training set and a test set. An initial model is constructed by a machine learning algorithm, and the initial model is trained by the training set to obtain a trained emotion recognition classification model. The emotion recognition classification model is tested on the test set to obtain corresponding test data.

[0016] Furthermore, the recognition efficiency is generated by the acquired test data , where the processing speed , Response time And accuracy After being treated as dimensionless, the corresponding weight coefficient is determined by referring to the hierarchical analysis method, and then according to the following method: ; The weight coefficient is , , , is a constant correction coefficient; if the recognition efficiency is If the efficiency threshold is lower than the threshold, the emotion recognition classification model should be optimized or retrained.

[0017] Furthermore, the multimodal emotion data collected in real time is preprocessed, and the corresponding feature data is extracted through feature engineering. The feature data is used as input, and the emotion recognition of the real-time data is performed using the trained emotion recognition classification model to obtain the corresponding recognition results; According to the correspondence between the recognition results and the response strategies, the response strategy library outputs the corresponding response strategies for the current teaching process, and adjusts and responds to the teaching behaviors of the teaching assistant robot according to the given response strategies.

[0018] Furthermore, after the teaching assistant robot responds according to the emotion recognition result, an observation cycle including several sub-cycles is set, and the feedback coefficient of the teaching activity process in each sub-cycle is , sort the sub-periods before and after the response according to the time axis and correspond one to one, according to the feedback coefficient The change in construction improvement factor , if the improvement coefficient If the improvement threshold is not exceeded, adjustment instructions are issued to the outside.

[0019] Furthermore, according to the feedback coefficient The change in construction improvement factor The way is as follows: ; in, For the i The median improvement of the sub-periods is is its mean value, is its qualified standard value; , is the number of sub-periods, and They are respectively before and after the teaching assistant robot responds i The feedback coefficient of the sub-cycle is and is the corresponding mean, weight coefficient: , ,and .

[0020] Furthermore, after receiving the adjustment instruction, the content of the response strategy is identified to increase the improvement coefficient within the observation period. As the optimization target, the response strategy is adjusted using a multi-objective optimization algorithm to obtain an optimized response strategy; Collect teaching process data during the observation period, such as test and examination scores, emotional feedback, and classroom participation; use them as sample data to train the machine learning algorithm to obtain the teaching feedback prediction model of the teaching assistant robot, and use the optimized response strategy as input, and the teaching feedback prediction model to predict its feedback effect.

[0021] An emotion recognition system for a teaching assistant robot includes a feedback data collection unit for collecting feedback data from students during class teaching, and generating a feedback coefficient in the current teaching activity process from the teaching feedback data. , if the feedback coefficient If the value is lower than the feedback threshold, a data collection instruction is issued to the outside; A collection path planning unit, in which a path planning algorithm is used to plan a data collection path in the teaching area, and multimodal emotion data is collected in the teaching area in sequence according to the data collection path, and the frequency of data collection is constrained, and the acquired data is aggregated to generate a multimodal emotion data set; The similarity analysis unit annotates the data in the multimodal emotion data set and generates an annotation accuracy based on the similarity analysis data after similarity analysis. If the marking accuracy If the value is higher than the precision threshold, the labeled multimodal emotion data is preprocessed to obtain preprocessed data; The response strategy output unit trains and acquires the emotion recognition classification model. After testing it, the recognition effect is verified by the acquired test data. If the recognition effect meets expectations, the multimodal emotion data collected in real time is used as input, and the trained emotion recognition classification model outputs the corresponding recognition result. The corresponding response strategy is matched for the teaching assistant robot based on the recognition result. The improvement optimization unit builds the improvement coefficient based on the feedback data changes during the observation period after the teaching assistant robot responds , if the improvement coefficient Without exceeding the improvement threshold, the response strategy is optimized and adjusted, and the optimized response strategy is verified by the teaching feedback prediction model to obtain the improvement coefficient again. , if the improvement coefficient If it is still lower than expected, the response strategy will be re-optimized.

[0022] (III) Beneficial effects The present invention provides an emotion recognition method and system for a teaching assistant robot, which has the following beneficial effects:

[0023] 1. Based on the feedback coefficient Evaluate the current teaching feedback effect to determine whether the current teaching feedback can meet expectations. When the current teaching status is poor or the teaching strategy is insufficient, timely discover and deal with it to ensure the smooth implementation of the teaching process.

[0024] 2. Divide the student distribution area into several sub-areas and then plan the data collection path, giving priority to data collection in important areas and areas with poor feedback, so as to prioritize processing and improvement.

[0025] 3. Evaluate the quality of data annotation based on the consistency between different annotation results. If the data annotation result is poor, re-annotate it to ensure the annotation result and make the emotion recognition result more accurate. Pre-process the acquired multimodal emotion data to ensure the data quality of the collected data.

[0026] 4. Based on recognition efficiency The performance of the emotion recognition classification model can be evaluated. When its performance cannot meet the expected effect, it can be optimized or retrained to ensure the recognition performance and recognition effect of the emotion recognition classification model.

[0027] 5. The trained emotion recognition and classification model recognizes the multimodal emotion data collected in real time, so that managers can timely grasp the current emotional state of students and adjust the current teaching status and strategy in time, so that the teaching process can get better feedback; the corresponding response strategy is output through the response strategy library to adjust and respond to the teaching assistant robot, so that students can get a better learning experience and better teaching feedback.

[0028] 6. By constraining the collection frequency and number of collection times in each sub-area, the amount of data collected can be matched with the teaching feedback status in each sub-area. When the teaching assistant robot responds and adjusts the teaching strategy for different sub-areas, there will be sufficient data support and coverage. For sub-areas with poor feedback and the corresponding students, the response of the teaching assistant robot can be more targeted.

[0029] 7. Based on the improvement coefficient Evaluate the current response effect to determine whether the current response is effective. If it fails to achieve the expected effect, you can respond again to obtain better teaching feedback.

[0030] 8. Generate improvement coefficients based on changes in feedback data during the observation period When the response of the teaching assistant robot fails to meet expectations, the response strategy can be adjusted and optimized, and the response strategy can be continuously iterated so that the teaching assistant robot can respond quickly based on the students’ emotions and other feedback data during the teaching process, thereby improving the teaching quality. By constructing a teaching feedback prediction model to predict the response strategy, the effectiveness of the response strategy can also be ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of the emotion recognition method for a teaching assistant robot according to the present invention; Figure 2 The figure is a schematic diagram of the structure of the emotion recognition system for the teaching assistant robot according to the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] See also Figure 1 The present invention provides an emotion recognition method for a teaching assistant robot, comprising: Step 1: Collect students’ feedback data during class teaching, and generate feedback coefficients in the current teaching process from the teaching feedback data , if the feedback coefficient If the value is lower than the feedback threshold, a data collection instruction is issued to the outside; The step 1 includes the following contents: Step 101: Pre-set a monitoring period and collect feedback data from students during class teaching, including students' participation in teaching activities. and interaction frequency , as well as facial expressions and postures, physiological reactions, peer interaction frequency, etc., and the obtained feedback data are aggregated to generate a teaching feedback data set; Step 102: Generate the feedback coefficient in the current teaching activity process from the teaching feedback data set ,in, On participation and interaction frequency Perform linear normalization and map the corresponding data values ​​to the interval In the following way: ; Weight coefficient: , and ;The weight coefficient can be obtained by referring to the hierarchical analysis method; For the i The degree of participation of students in teaching activities, is the qualified standard value of participation, is the mean of participation; For the i The frequency of interaction between students in teaching activities, is the qualified standard value of interaction frequency, is the mean of interaction frequency; Based on historical data and expectations of teaching feedback, the feedback threshold is set in advance; if the feedback coefficient If it is lower than the feedback threshold, it means that the current teaching process has not achieved the expected effect. Under the condition of obtaining students' emotions, it is necessary to adjust the current teaching content or teaching method. At this time, a data collection instruction for multimodal emotional data is issued to the outside. When using, combine the contents in steps 101 and 102: After using the teaching assistant robot to intervene in the current teaching activities, the teaching feedback data of each student is collected and the feedback coefficient is generated. , according to the feedback coefficient obtained Evaluate the current teaching feedback effect to determine whether the current teaching feedback can meet expectations and whether students are in a good learning state, so that when the current teaching state is poor or the teaching strategy is insufficient, it can be discovered and dealt with in a timely manner to ensure the smooth implementation of the teaching process.

[0034] Combined with the above application and the contents of the prior art: When applying educational robots in teaching activities, students' emotional changes can provide feedback on the current teaching status and teaching results. When teachers identify such emotional changes, they can respond and provide feedback on the current teaching process, and adjust and improve the current teaching results. Especially when students in the teaching area have low interactive participation and low interaction frequency, if the teaching and response strategies of the teaching assistant robots cannot be adjusted in time, the current teaching effect may decline rapidly; but when existing teaching assistant robots identify students' emotions and respond accordingly, they usually need to send the emotion recognition results to the administrators, who will give adjusted response strategies. This can easily lead to the teaching assistant robots responding slowly after completing emotion recognition, and failing to quickly adjust the current poor teaching feedback.

[0035] Step 2: A data collection path is planned by a path planning algorithm in the teaching area, and multimodal emotion data is collected in the teaching area in sequence according to the data collection path, and the frequency of data collection is constrained, and the acquired data is aggregated to generate a multimodal emotion data set; The step 2 includes the following contents: Step 201: after receiving the data collection instruction in the teaching stage, the student distribution area is grid-divided into a plurality of sub-areas, and after constructing an electronic map of the student distribution area, each sub-area is marked on the electronic map; Obtain the feedback coefficient and position information in each sub-area, use the path planning algorithm to plan the path for each sub-area, obtain the corresponding data collection path, and constrain the order of data collection; When in use, when it is necessary to collect students' emotional feedback and other feedback data, the data collection path is planned by dividing the student distribution area into several sub-areas, and data collection is prioritized for important areas and areas with poor feedback, so as to prioritize processing and improvement.

[0036] Step 202: After determining the data collection sequence, the data collection type is limited. When collecting multimodal emotion data in each sub-area, the frequency of data collection is constrained. The constraint method is as follows: ; in, n is the number of nodes collecting multimodal emotion data, It is i The collection node to the j The time interval of each collection node, is the average value of the time interval, is the feedback coefficient; Data is collected in the sub-area at the data collection nodes that meet the constraints, and the acquired data is aggregated to generate a multimodal emotion data set; specifically, visual data: using a high-definition camera to collect the user's facial images and videos; audio data: collecting the user's voice signal through a high-quality microphone and sampling in different environments; text data: collecting chat records and input text generated by the user during the interaction with the teaching assistant robot.

[0037] When using, combine the contents in steps 201 to 202: When data collection is actually needed, the collection frequency and number of collections in each sub-area are constrained so that the amount of data collected matches the teaching feedback status in each sub-area. When the teaching assistant robot responds and adjusts the teaching strategy for different sub-areas, there will be sufficient data support and coverage, making the response of the teaching assistant robot more targeted for sub-areas with poor feedback and the corresponding students.

[0038] Step 3: Label the data in the multimodal emotion data set, and after similarity analysis, generate the labeling accuracy based on the similarity analysis data. If the marking accuracy If the value is higher than the precision threshold, the labeled multimodal emotion data is preprocessed to obtain preprocessed data; The step three includes the following contents: Step 301: After obtaining the data in the multimodal emotion data set, the selected emotion psychology experts annotate the data in detail, including emotion categories and intensities, and use the annotated data as the first annotated set; multiple other annotators annotate the same data, obtain the annotated data and use it as the second annotated set; Perform similarity analysis on the two annotation results in the first annotation set and the second annotation set, obtain the similarity data between the annotations of the annotator and the expert, and then generate a corresponding similarity data set; When in use, by performing similarity analysis on different annotation results, the quality of data annotation can be evaluated based on the consistency between different annotation results. If the data annotation result is poor, it can be re-annotated to ensure the annotation result and make the emotion recognition result more accurate.

[0039] Step 302: Under dimensionless conditions, generate a labeling accuracy from the similarity data in the similarity data set , as follows: ; in, n is the number of annotators, It is i The similarity of data annotation between annotators and experts, is the average similarity value, is the qualified target value of similarity; weight coefficient: , ,and ; Preset the accuracy threshold based on historical data and management expectations of annotation effects; if the annotation accuracy is If the current annotation accuracy is lower than the precision threshold, it means that the current annotation accuracy is insufficient and needs to be re-annotated. If it is higher than the precise threshold, it means that the current annotation effect is good. In this case, a preprocessing instruction is issued to the outside. After receiving the preprocessing instructions, the annotated multimodal emotion data is preprocessed. The preprocessing scheme includes: filtering, using Kalman filtering, low-pass filtering and other technologies to remove noise and artifacts in the signal; feature extraction, extracting time domain features (such as average heart rate, standard deviation) and frequency domain features (such as power spectral density); text cleaning, removing special characters, stop words, etc.; word segmentation and word vector representation, using natural language processing technology (such as Word2Vec, GloVe, BERT) to convert text into word vectors.

[0040] When used, combine the contents in steps 301 and 302: Based on the similarity analysis, the annotation accuracy is generated from the obtained similarity data , according to the marking accuracy A comprehensive evaluation of the current data annotation results can ensure the accuracy of data annotation. By preprocessing the acquired multimodal emotion data, the data quality of the collected data can also be guaranteed.

[0041] Step 4: Train and acquire the emotion recognition classification model. After testing it, verify its recognition effect with the acquired test data. If the recognition effect meets expectations, the multimodal emotion data collected in real time is used as input, and the trained emotion recognition classification model outputs the corresponding recognition result. According to the recognition result, the corresponding response strategy is matched for the teaching assistant robot. The step 4 includes the following contents: Step 401: After obtaining the preprocessed data, select part of the data from the preprocessed data as sample data, extract and fuse the features of the sample data, divide the sample data into a training set and a test set, build an initial model by a machine learning algorithm, train the initial model by the training set, and obtain a trained emotion recognition classification model; The emotion recognition classification model was tested on the test set to verify its reliability. After several tests, the corresponding test data was obtained, including processing speed, response time, and accuracy. Step 402: Generate recognition efficiency from the acquired test data , where the processing speed , Response time And accuracy After being treated as dimensionless, the corresponding weight coefficient is determined by referring to the hierarchical analysis method, and then according to the following method: ; The weight coefficient is, , , , is a constant correction coefficient whose value falls within ; Pre-set efficiency thresholds based on historical data and expectations of recognition results; If the recognition efficiency is If the obtained recognition efficiency is lower than the efficiency threshold, it means that the current recognition result is poor and the emotion recognition classification model needs to be optimized or retrained. If the obtained recognition efficiency is not lower than the efficiency threshold, it means that the current recognition result can meet expectations. At this time, the training of the emotion recognition classification model is completed. When using, after preprocessing the multimodal emotion data, use it as sample data to train the emotion recognition classification model, and test the emotion recognition classification model through the test set, and generate the recognition efficiency index based on the obtained test data. , according to the recognition efficiency The performance of the emotion recognition classification model can be evaluated. When its performance cannot meet the expected effect, it can be optimized or retrained to ensure the recognition performance and recognition effect of the emotion recognition classification model.

[0042] Step 403: pre-process the multimodal emotion data collected in real time, extract corresponding feature data through feature engineering, use the feature data as input, use the trained emotion recognition classification model to perform emotion recognition on the real-time data, and obtain corresponding recognition results; and feed back the obtained recognition results to the user through text, voice, etc.; Step 404: after constructing or collecting a number of teaching response strategies, a response strategy library is generated after aggregation; according to the correspondence between the recognition results and the response strategies, the response strategy library outputs the corresponding response strategy for the current teaching process, and the teaching behavior of the teaching assistant robot is adjusted and responded according to the given response strategy, so that the teaching assistant robot can take the corresponding operation mode according to the emotion recognition results; When used, combine the contents in steps 401 to 404: After the construction and training of the emotion recognition classification model is completed, the trained emotion recognition classification model will recognize the multimodal emotion data collected in real time. After obtaining the recognition results, the management staff can timely grasp the current emotional state of the students, so as to adjust the current teaching status and strategy in time and get better feedback on the teaching process. Furthermore, by outputting corresponding response strategies through the response strategy library and adjusting and responding to the teaching assistant robot, the teaching assistant robot can grasp the current teaching dynamics in a timely manner during the teaching process, so that students can get a better learning experience and obtain better teaching feedback.

[0043] Step 5: After the teaching assistant robot responds, construct the improvement coefficient based on the feedback data changes during the observation period , if the improvement coefficient Without exceeding the improvement threshold, the response strategy is optimized and adjusted, and the optimized response strategy is verified by the teaching feedback prediction model to obtain the improvement coefficient again. , if the improvement coefficient If it is still lower than expected, the response strategy will be re-optimized; The step five includes the following contents: Step 501: After the teaching assistant robot responds based on the emotion recognition result, an observation period including several sub-periods is set, and the feedback coefficients in the teaching activity process in each sub-period are , sort the sub-periods before and after the response according to the time axis and correspond one to one, according to the feedback coefficient The change in construction improvement factor , as follows: ; in, For the i The median improvement of the sub-periods, is its mean value, is its qualified standard value; , is the number of sub-periods, and They are respectively before and after the teaching assistant robot responds i The feedback coefficient of the sub-cycle is and is the corresponding mean, weight coefficient: , ,and ;The weight coefficient value remains consistent with the previous value; Based on historical data and the expected improvement of teaching feedback, the improvement threshold is set in advance; if the improvement coefficient If the improvement threshold is not exceeded, it means that after the teaching assistant robot responds, the improvement of the teaching feedback cannot achieve the expected effect. At this time, an adjustment instruction is issued to the outside. When in use, after completing emotion recognition and the assistant robot responds, the feedback coefficient The change in construction improvement factor , according to the improvement factor It can evaluate the current response effect and determine whether the current response is effective. If it fails to achieve the expected effect, it can respond again to obtain better teaching feedback.

[0044] Step 502: After receiving the adjustment instruction, identify the content of the response strategy to increase the improvement coefficient within the observation period. As the optimization target, the response strategy is adjusted using a multi-objective optimization algorithm to obtain an optimized response strategy; Collect teaching process data during the observation period, such as test and examination scores, emotional feedback, and classroom participation, etc.; use them as sample data to train the machine learning algorithm to obtain the teaching feedback prediction model of the teaching assistant robot, use the optimized response strategy as input, and use the teaching feedback prediction model to predict its feedback effect, and then obtain the improvement coefficient again from the obtained prediction data , if the improvement coefficient If it is still lower than expected, it will be optimized again; When using, combine the contents in steps 501 and 502: After completing the training of the emotion recognition classification model, the student emotion recognition during the teaching process, and the teaching assistant robot making targeted responses, the improvement coefficient is generated based on the feedback data changes during the observation period. When the response of the teaching assistant robot fails to meet expectations, by adjusting and optimizing the response strategy and continuously iterating the response strategy, the teaching assistant robot can quickly respond based on the students' emotions and other feedback data during the teaching process, thereby improving the teaching quality. At the same time, by constructing a teaching feedback prediction model to predict the response strategy, the effectiveness of the response strategy can also be ensured.

[0045] The Analytic Hierarchy Process (AHP) is a decision-making method that decomposes decision-related elements into levels such as goals, criteria, and plans, and conducts qualitative and quantitative analysis on this basis. It is particularly suitable for dealing with target systems with hierarchical and staggered evaluation indicators, and when the target value is difficult to describe quantitatively, the AHP is an effective decision-making tool.

[0046] The core of the hierarchical analysis method is to decompose the decision problem into multiple levels to form a hierarchical structure, which usually includes the target level, the criterion level, the sub-criterion level and the solution level. By solving the eigenvector of the judgment matrix, the priority weight of each element of each level to an element of the previous level is obtained, and the final weight of each alternative solution to the total goal is recursively merged by the weighted sum method, so as to find the optimal solution.

[0047] See also Figure 2 The present invention provides an emotion recognition system for a teaching assistant robot, comprising: Feedback data collection unit collects students' feedback data during class teaching, and generates feedback coefficients in the current teaching process based on teaching feedback data , if the feedback coefficient If the value is lower than the feedback threshold, a data collection instruction is issued to the outside; A collection path planning unit, in which a path planning algorithm is used to plan a data collection path in the teaching area, and multimodal emotion data is collected in the teaching area in sequence according to the data collection path, and the frequency of data collection is constrained, and the acquired data is aggregated to generate a multimodal emotion data set; The similarity analysis unit annotates the data in the multimodal emotion data set and generates an annotation accuracy based on the similarity analysis data after similarity analysis. If the marking accuracy If the value is higher than the precision threshold, the labeled multimodal emotion data is preprocessed to obtain preprocessed data; The response strategy output unit trains and acquires the emotion recognition classification model. After testing it, the recognition effect is verified by the acquired test data. If the recognition effect meets expectations, the multimodal emotion data collected in real time is used as input, and the trained emotion recognition classification model outputs the corresponding recognition result. The corresponding response strategy is matched for the teaching assistant robot based on the recognition result. The improvement optimization unit builds the improvement coefficient based on the feedback data changes during the observation period after the teaching assistant robot responds , if the improvement coefficient Without exceeding the improvement threshold, the response strategy is optimized and adjusted, and the optimized response strategy is verified by the teaching feedback prediction model to obtain the improvement coefficient again. , if the improvement coefficient If it is still lower than expected, the response strategy will be re-optimized.

[0048] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0049] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0050] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0051] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only some logical function divisions. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0052] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0053] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0054] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0055] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An emotion recognition method for a teaching assistant robot, characterized in that: include, Collect students' feedback data during class teaching, and generate feedback coefficients in the current teaching process based on the teaching feedback data , if the feedback coefficient If the value is lower than the feedback threshold, a data collection instruction is issued to the outside; The path planning algorithm plans a data collection path in the teaching area, and multimodal emotion data is collected in the teaching area in turn according to the data collection path, and the frequency of data collection is constrained, and the acquired data is aggregated to generate a multimodal emotion data set; Label the data in the multimodal emotion data set, and after similarity analysis, generate the labeling accuracy based on the similarity analysis data. If the marking accuracy If the value is higher than the precision threshold, the labeled multimodal emotion data is preprocessed to obtain preprocessed data; Train and acquire the emotion recognition classification model. After testing it, verify its recognition effect with the acquired test data. If the recognition effect meets expectations, the multimodal emotion data collected in real time is used as input, and the trained emotion recognition classification model outputs the corresponding recognition result. According to the recognition result, the corresponding response strategy is matched for the teaching assistant robot. After the teaching assistant robot responds, the improvement coefficient is constructed based on the feedback data changes during the observation period. , if the improvement coefficient Without exceeding the improvement threshold, the response strategy is optimized and adjusted, and the optimized response strategy is verified by the teaching feedback prediction model to obtain the improvement coefficient again. , if the improvement coefficient If it is still lower than expected, the response strategy will be re-optimized.

2. The emotion recognition method for a teaching assistant robot according to claim 1, characterized in that: Collect students' feedback data during class teaching during the monitoring period, and aggregate the acquired feedback data to generate a teaching feedback data set; Generate feedback coefficients from teaching feedback data set , among which, the participation and interaction frequency Perform linear normalization and map the corresponding data values ​​to the interval In the following way: ； Weight coefficient: , and ; For the i The degree of participation of students in teaching activities, is the qualified standard value of participation, is the mean of participation; For the i The frequency of interaction between students in teaching activities, is the qualified standard value of interaction frequency, is the mean interaction frequency.

3. The emotion recognition method for a teaching assistant robot according to claim 2, characterized in that: After receiving the data collection instruction, the student distribution area is divided into several sub-areas, and after constructing an electronic map of the student distribution area, each sub-area is marked on the electronic map; The feedback coefficient and position information in each sub-area are obtained, and the path planning algorithm is used to plan the path for each sub-area to obtain the corresponding data collection path.

4. The emotion recognition method for a teaching assistant robot according to claim 3, characterized in that: After determining the data collection order, the data collection types are limited. When collecting multimodal emotion data in each sub-area, the frequency of data collection is constrained. Data is collected in the sub-area at the data collection nodes that meet the constraints. The constraints are as follows: ； in, n is the number of nodes collecting multimodal emotion data, It is i The collection node to the j The time interval of each collection node, is the average value of the time interval, is the feedback coefficient.

5. The emotion recognition method for a teaching assistant robot according to claim 4, characterized in that: After obtaining the data in the multimodal emotion data set, the selected experts will annotate the data in detail, and the annotated data will be used as the first annotated set; multiple other annotators will annotate the same data, obtain the annotated data and use it as the second annotated set; A similarity analysis is performed on the two annotation results in the first annotation set and the second annotation set, and a corresponding similarity data set is generated after obtaining similarity data.

6. The emotion recognition method for a teaching assistant robot according to claim 5, characterized in that: Under dimensionless conditions, the annotation accuracy is generated from the similarity data in the similarity data set. , as follows: ； Weight coefficient: , ,and , n is the number of annotators, It is i The similarity of data annotation between annotators and experts, is the average similarity value, is the qualified target value for similarity.

7. The emotion recognition method for a teaching assistant robot according to claim 6, characterized in that: After obtaining the preprocessed data, select part of the data from the preprocessed data as sample data, divide the sample data into a training set and a test set, build an initial model using a machine learning algorithm, train the initial model using the training set, obtain the trained emotion recognition classification model, test the emotion recognition classification model on the test set, and obtain corresponding test data.

8. The emotion recognition method for a teaching assistant robot according to claim 7, characterized in that: Generate recognition efficiency from the acquired test data , where the processing speed , Response time And accuracy After being treated as dimensionless, the corresponding weight coefficient is determined by referring to the hierarchical analysis method, and then according to the following method: ； The weight coefficient is, , , , is the constant correction factor; If the recognition efficiency is If the efficiency threshold is lower than the threshold, the emotion recognition classification model should be optimized or retrained.

9. The emotion recognition method for a teaching assistant robot according to claim 8, characterized in that: Preprocess the multimodal emotion data collected in real time, extract the corresponding feature data through feature engineering, use the feature data as input, use the trained emotion recognition classification model to perform emotion recognition on the real-time data, and obtain the corresponding recognition results; According to the correspondence between the recognition results and the response strategies, the response strategy library outputs the corresponding response strategies for the current teaching process, and adjusts and responds to the teaching behaviors of the teaching assistant robot according to the given response strategies.

10. The emotion recognition method for a teaching assistant robot according to claim 9, characterized in that: After the teaching assistant robot responds based on the emotion recognition results, an observation cycle consisting of several sub-cycles is set, and the feedback coefficient of the teaching activity in each sub-cycle is , sort the sub-periods before and after the response according to the time axis and correspond one to one, according to the feedback coefficient The change in construction improvement factor , if the improvement coefficient If the improvement threshold is not exceeded, adjustment instructions are issued to the outside.

11. The emotion recognition method for a teaching assistant robot according to claim 10, characterized in that: Based on the feedback factor The change in construction improvement factor The way is as follows: ； in, For the i The median improvement of the sub-periods is is its mean value, is its qualified standard value; , is the number of sub-periods, and They are respectively before and after the teaching assistant robot responds i The feedback coefficient of the sub-cycle is and is the corresponding mean, weight coefficient: , ,and .

12. The emotion recognition method for a teaching assistant robot according to claim 11, characterized in that: After receiving the adjustment instruction, identify the content of the response strategy to increase the improvement coefficient within the observation period As the optimization target, the response strategy is adjusted using a multi-objective optimization algorithm to obtain an optimized response strategy; The teaching process data within the observation period is collected and used as sample data to train the machine learning algorithm to obtain the teaching feedback prediction model of the teaching assistant robot. The optimized response strategy is used as input, and the teaching feedback prediction model predicts its feedback effect.

13. An emotion recognition system for a teaching assistant robot, characterized in that: include, Feedback data collection unit collects students' feedback data during class teaching, and generates feedback coefficients in the current teaching process based on teaching feedback data , if the feedback coefficient If the value is lower than the feedback threshold, a data collection instruction is issued to the outside; A collection path planning unit, in which a path planning algorithm is used to plan a data collection path in the teaching area, and multimodal emotion data is collected in the teaching area in sequence according to the data collection path, and the frequency of data collection is constrained, and the acquired data is aggregated to generate a multimodal emotion data set; The similarity analysis unit annotates the data in the multimodal emotion data set and generates an annotation accuracy based on the similarity analysis data after similarity analysis. If the marking accuracy If the value is higher than the precision threshold, the labeled multimodal emotion data is preprocessed to obtain preprocessed data; The response strategy output unit trains and acquires the emotion recognition classification model. After testing it, the recognition effect is verified by the acquired test data. If the recognition effect meets expectations, the multimodal emotion data collected in real time is used as input, and the trained emotion recognition classification model outputs the corresponding recognition result. The corresponding response strategy is matched for the teaching assistant robot based on the recognition result. The improvement optimization unit builds the improvement coefficient based on the feedback data changes during the observation period after the teaching assistant robot responds , if the improvement coefficient Without exceeding the improvement threshold, the response strategy is optimized and adjusted, and the optimized response strategy is verified by the teaching feedback prediction model to obtain the improvement coefficient again. , if the improvement coefficient If it is still lower than expected, the response strategy will be re-optimized.

Images