A teaching system based on AIGC classroom behavior feedback

Abstract

The present application discloses a teaching system based on AIGC classroom behavior feedback, which relates to the fields of computer artificial intelligence and education, and in particular to an AI-based virtual teaching system, which uses artificial intelligence technology to realize the teaching of a virtual private teacher; the teaching interface is divided into two parts, one part is a virtual teacher, which can be a cartoon character, a social celebrity, etc., and the other part is a visual teaching interface, which displays the teaching content and assists users to better understand the teaching content; and the system is divided into a MOOC teaching part and a 1v1 question-and-answer part, each of which is further divided into a student end, an administrator end and a database, each of which plays a different role; a feedback mechanism is added to collect audience feedback information such as eye movement information and facial information to analyze their concentration and knowledge points that have not been mastered; and a speech end judgment mechanism is proposed to judge whether the audience has ended the conversation and feedback the corresponding ending words to the audience.

Description

Technical Field

[0001] The present invention relates to the fields of computer artificial intelligence and education, and in particular to an AIGC classroom behavior feedback teaching system that implements teaching by a virtual private teacher through artificial intelligence technology. Background Art

[0002] With the continuous advancement of technology, the application of artificial intelligence in education has gradually attracted people's attention. In today's society, the importance of education is increasing day by day. Many parents strive for better learning resources for their children, such as hiring tutors or one-on-one tutoring. However, finding the right qualified teachers and providing face-to-face teaching is a considerable task.

[0003] Against this backdrop, a teaching system based on AIGC classroom behavior feedback has emerged as a development opportunity. This system acts as a 24 / 7 personal virtual teacher, answering questions across various subjects, teaching multidisciplinary courses, and providing personalized learning support. This new educational approach is expected to reduce academic pressure on students and alleviate parental anxiety, creating a more flexible and flexible learning environment for students. Summary of the Invention

[0004] In response to the problems existing in the existing technology, the present invention provides a teaching system based on AIGC classroom behavior feedback, which divides the teaching interface into two parts. One part is a virtual teacher, which can be a cartoon character, a social celebrity, etc., and the other part is a visual teaching interface, which displays the teaching content and helps users better understand the teaching content. Moreover, the system is divided into a MOOC teaching part and a 1v1 teaching part, each of which is further divided into a student side, an administrator side and a database, each of which plays a different role.

[0005] It further includes MOOC teaching part and 1v1 teaching part:

[0006] The MOOC teaching part is divided into the student side, the administrator side and the database. The student side users log in, study the MOOC and choose a virtual teacher. The administrator side is responsible for retrieving database resources, returning them to the student side, collecting student information from the student side and storing it in the database, and analyzing user behavior based on the relevant information.

[0007] The 1v1 teaching part is divided into the student side, the administrator side and the database. The student side logs in, selects the 1v1 teaching part, and expresses their doubts. The administrator side is responsible for collecting and analyzing the student side's voice and other information, and returns the question-and-answer video after processing it with the large language model and AI model. It also analyzes user behavior, determines the end of the question, and returns the corresponding conclusion.

[0008] It further includes the database part:

[0009] Create a teacher database, a teaching material database, and a student behavior database. The teacher database includes videos of virtual teachers teaching; the teaching material database includes teacher lecture-related materials and teaching visualization materials to assist teaching; the student behavior database includes collected student-related information.

[0010] Preferably, the MOOC teaching part of the present invention specifically includes the following steps:

[0011] Step S1: Collect the teacher's facial information and voice information when giving a lecture.

[0012] Step S2: Perform data preprocessing on the facial information and voice information in step S1.

[0013] Step S3: Extract features from the facial information in step S2, and extract features from the voice information and convert them into text information:

[0014] For facial information, extract its facial expression information;

[0015] For voice information, its pitch information is extracted.

[0016] Step S4: The feature information in step S3 is stored in the teacher database.

[0017] Step S5: Use programming technology to create teaching visualization videos to assist in understanding knowledge points; collect course-related knowledge point materials and store them in the teaching material database.

[0018] Step S6: The user logs in on the student side, selects the "MOOC teaching" section, selects a specific course and selects the image of the virtual teacher.

[0019] Step S7: After the administrator receives the "MOOC teaching" signal in step S6, it receives the virtual teacher image selected by the user, retrieves the data information of the relevant teachers in the teacher database, and fits it through AIGC to form a virtual teacher teaching video; and retrieves the information of the relevant courses in the teaching resource database, returns the virtual teacher teaching video and the visual teaching video to the student interface, and returns the course-related knowledge point information to the user in PDF format.

[0020] Step S8: After the user starts watching the video returned in step S7, the video information is sliced ​​every ten seconds according to the time, the user's facial information and eye movement information are collected, stored in the student behavior database, and data preprocessing and feature extraction are performed on them:

[0021] For eye movement information, extract the eye movement trajectory, pupil changes and gaze point changes;

[0022] For facial information, extract its facial expression information and the degree of mouth opening and closing.

[0023] Step S9: The user information obtained in step S8 is analyzed through machine learning to obtain a model for evaluating whether the user understands and a model for evaluating the user's concentration. The parts that are judged not to be understood are marked on the knowledge point information in the teaching material database, and a class report is generated based on the concentration and understanding ratio. The class report is stored in the student behavior database and fed back to the user at the end of the class.

[0024] Preferably, the 1v1 teaching part of the present invention includes the following steps:

[0025] Step W1: The user logs in on the student side, selects the “1v1 teaching” section, and chooses the image of the virtual teacher.

[0026] Step W2: After receiving the "1v1 teaching" signal in step W1, the administrator side accepts the virtual teacher image selected by the user, slices the video every ten seconds according to time, and transmits it to the administrator side, and starts receiving the user's voice information, facial information and eye movement information.

[0027] Step W3: Perform data preprocessing on the voice information, facial information and eye movement information obtained in step W2, and convert the voice information into text information.

[0028] Step W4: Perform feature extraction on the voice information, facial information, eye movement information, and text information processed in step W3:

[0029] For facial information, extract lip movement information, facial expression information and mouth opening degree;

[0030] For eye movement information, extract the eye movement trajectory and pupil changes;

[0031] For pitch information, extract its frequency contour, pitch contour and pitch feature vector;

[0032] Extract contextual sentiment from text information;

[0033] After completion, proceed to step W6.

[0034] Step W5: The text information obtained in S3 is transmitted to the large language model, and the corresponding answer is returned. It is put into the AI ​​model together with the image of the virtual teacher obtained in step S2 to generate a virtual teacher video, which is returned to the student end.

[0035] Step W6: The data extracted from features in W4 is put into machine learning to form a model for determining the end of the question. If the conversation is judged to be over, the corresponding ending statement is returned; if the conversation is not over, the question continues to be received.

[0036] Furthermore, step S9 specifically includes the following steps:

[0037] Step S91: The eye movement trajectory and pupil change characteristics of the eye movement information obtained in step S8 are trained and predicted using a convolutional neural network (CNN) model, and output whether there is any doubt about the knowledge point.

[0038] Step S92: The facial expression information features of the facial information obtained in step S8 are trained and predicted using SVM, and output whether there is any doubt about the knowledge point.

[0039] Step S93: The mouth opening and closing degree feature of the facial information obtained in step S8 is trained and predicted using SVM, and outputs whether there is any doubt about the knowledge point.

[0040] Step S94: Put the characteristic values ​​output from steps S91, S92, and S93 into the following formula to make a final judgment. The formula is as follows:

[0041]

[0042] Where X is a vector feature matrix, each column represents an eigenvalue (a total of 3 eigenvalues), and each row represents a sample, forming an n*3 matrix. For each sample feature, label it (1 for understood / 0 for not understood) and build the matrix Y;

[0043] θ i To assign a weight to each feature value (find the best θ through the training process i value), ∑ i＝1 n(θ1Y i1 +θ2Y i2 +θ3Y i3 ),∑ i＝1 n(Y ji ),∑ j＝1 3(Q j ), P1, P2: If the result reaches the preset threshold (the best threshold is found through the training process) or above, it is considered "understood" and marked as 1; if it does not reach the preset threshold, it is considered "not understood" and marked as 0;

[0044] P1 is the result of judging whether the speaker understands the sample, Q j P is the result of judging whether the audio is understood or not calculated based on the eigenvalues ​​of each column, and P2 is the result of judging whether the audio is understood or not calculated based on each eigenvalue;

[0045] The intersection of the two results is taken, that is, when both are judged as "understood", the result is "understood", and in other cases it is judged as "not understood". If it is judged as not understood, go to step S95.

[0046] Step S95: extract the sound information from the video information, convert it into text, search for relevant knowledge point information in the teaching material database, mark it, and then store it in the database.

[0047] Step S96: Use computer vision technology to estimate the gaze point change feature information of the eye movement information in step S8 and draw a radar chart, and judge the user's concentration by analyzing the stability, duration and position of the gaze point on the screen.

[0048] Step S97: Combine the results of step S95 and step S96, summarize them into a report, and feed back to the student end after class.

[0049] Furthermore, step W6 is specifically as follows:

[0050] Step W61: Use the facial expression information, eye movement trajectory, and pupil changes obtained in step W4 to fit the emotion analysis model using SVM, train the model using labeled training data, simulate the target's emotions, and output whether the speech has ended.

[0051] Step W62: Label the lip movements and still frames based on the timestamp of the video data using the lip information obtained in step W4, create a training data set, select a CNN model, and train the labeled data training set to obtain a model that determines whether the lips are moving and whether the speech has ended.

[0052] Step W63: The frequency contour, pitch contour and pitch feature vector of the tone obtained in step W4 are passed through an SVM training model, and the training data with labels are used for training as a basis for judgment.

[0053] Step W64: The results of the model output in steps W61 to W63 are used to make a final judgment using the following formula:

[0054] p final =p 唇动 ∩Σ i∈S w i *p i

[0055] in The set S is {emotion, tone};

[0056] Will Through cost sensitivity analysis, if the preset threshold is reached, it is recorded as 1, if not reached, it is recorded as 0, the performance and cost of the model under different thresholds are estimated, and the optimal threshold is selected; in the above formula:

[0057] p 唇动Refers to the result obtained by the lip movement information model, the result is 1 (end of speaking) or 0 (not end of speaking), p 情绪 Refers to the result of the sentiment analysis model, the result is 1 or 0, p 音调 The result of the pitch information model is 1 or 0; w 情绪 It refers to the ratio of the F1 score of the sentiment analysis model to the sum of the F1 scores of the sentiment analysis model and the tone information model. 音调 The ratio of the F1 score of the tone information model to the sum of the F1 scores of the sentiment analysis model and the tone information model;

[0058] Finally, determine whether the speech is finished based on the formula result. If the speech is finished, go to step W65. If not, continue collecting.

[0059] Step W65: Analyze the text information and contextual emotions obtained in step W4 through a large language model to derive the emotional tendency in the context and understand the expression and intention in the context, give corresponding feedback on the end of the speech, and adjust the other party's emotions. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments.

[0061] Figure 1 4 is a structural diagram of an AIGC-based classroom behavior feedback teaching system according to an embodiment of the present invention.

[0062] Figure 2 It is a flowchart of step S9 in an embodiment of the present invention.

[0063] Figure 3 It is a flowchart of step W6 in an embodiment of the present invention.

[0064] The present invention will be further described below in conjunction with the accompanying drawings:

[0065] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0066] In addition, the terms "include," "comprises," and "has," and any variations thereof, are intended to cover a non-exclusive inclusion.

[0067] An embodiment of the present invention provides an AIGC-based classroom behavior feedback teaching system, which divides the teaching interface into two parts: one part is a virtual teacher, which can be a cartoon character, a social celebrity, etc., and the other part is a visual teaching interface that displays teaching content to help users better understand the teaching content; and the system is divided into a MOOC teaching part and a 1v1 teaching part, each of which is further divided into a student side, an administrator side and a database, respectively performing different functions.

[0068] It further includes MOOC teaching part and 1v1 teaching part:

[0069] The MOOC teaching part is divided into the student side, the administrator side and the database. The student side users log in with their registered account and password to study MOOC and select a virtual teacher. The administrator side is responsible for retrieving database resources, returning them to the student side, collecting student information from the student side and storing it in the database, and analyzing user behavior based on the relevant information.

[0070] The 1v1 teaching part is divided into the student side, the administrator side and the database. On the student side, students log in with their account and password, select the 1v1 teaching part, and express their doubts; the administrator side is responsible for collecting and analyzing the student side's voice and other information, and returns the question-and-answer video after processing it with the large language model and AI model. It also analyzes user behavior, determines that the question is over, and returns the corresponding conclusion.

[0071] Further includes a database part

[0072] Create a teacher database, a teaching material database, and a student behavior database. The teacher database includes videos of virtual teachers teaching; the teaching material database includes teacher lecture-related materials and teaching visualization materials to assist teaching; the student behavior database includes collecting student-related information.

[0073] Preferably, the MOOC teaching part of the present invention specifically includes the following steps:

[0074] Step S1: Collect the teacher's facial information and voice information when giving a lecture.

[0075] Step S2: Perform data preprocessing on the facial information and voice information in step S1.

[0076] Step S3: Extract features from the facial information in step S2, and extract features from the voice information and convert them into text information:

[0077] For facial information, extract its facial expression information;

[0078] For voice information, its pitch information is extracted.

[0079] Step S4: The feature information in step S3 is stored in the teacher database.

[0080] Step S5: Use programming technology to create teaching visualization videos to assist in understanding knowledge points; collect course-related knowledge point materials and store them in the teaching material database.

[0081] Step S6: The user logs in on the student side, selects the "MOOC teaching" section, selects a specific course and selects the image of the virtual teacher.

[0082] Step S7: After the administrator receives the "MOOC teaching" signal in step S6, it receives the virtual teacher image selected by the user, retrieves the data information of the relevant teachers in the teacher database, and fits it through AIGC to form a virtual teacher teaching video; and retrieves the information of the relevant courses in the teaching resource database, returns the virtual teacher teaching video and the visual teaching video to the student interface, and returns the course-related knowledge point information to the user in PDF format.

[0083] Step S8: After the user starts watching the video returned in step S7, the video information is sliced ​​every ten seconds according to the time, the user's facial information and eye movement information are collected, stored in the student behavior database, and data preprocessing and feature extraction are performed on them:

[0084] For eye movement information, extract the eye movement trajectory, pupil changes and gaze point changes;

[0085] For facial information, extract its facial expression information and the degree of mouth opening and closing.

[0086] Step S9: The user information obtained in step S8 is analyzed through machine learning to obtain a model for evaluating whether the user understands and a model for evaluating the user's concentration. The parts that are judged not to be understood are marked on the knowledge point information in the teaching material database, and a class report is generated based on the concentration and understanding ratio. The class report is stored in the student behavior database and fed back to the user at the end of the class.

[0087] Preferably, the 1v1 teaching part of the present invention includes the following steps:

[0088] Step W1: The user logs in on the student side, selects the “1v1 teaching” section, and chooses the image of the virtual teacher.

[0089] Step W2: After receiving the "1v1 teaching" signal in step W1, the administrator side accepts the virtual teacher image selected by the user, slices the video every ten seconds according to time, and transmits it to the administrator side, and starts receiving the user's voice information, facial information and eye movement information.

[0090] Step W3: Perform data preprocessing on the voice information, facial information and eye movement information obtained in step W2, and convert the voice information into text information.

[0091] Step W4: Perform feature extraction on the voice information, facial information, eye movement information, and text information processed in step W3:

[0092] For facial information, extract lip movement information, facial expression information and mouth opening degree;

[0093] For eye movement information, extract the eye movement trajectory and pupil changes;

[0094] For pitch information, extract its frequency contour, pitch contour and pitch feature vector;

[0095] Extract contextual sentiment from text information;

[0096] After completion, proceed to step W6.

[0097] Step W5: The text information obtained in S3 is transmitted to the large language model, and the corresponding answer is returned. It is put into the AI ​​model together with the image of the virtual teacher obtained in step S2 to generate a virtual teacher video, which is returned to the student end.

[0098] Step W6: The data extracted from features in W4 is put into machine learning to form a model for determining the end of the question. If the conversation is judged to be over, the corresponding ending statement is returned; if the conversation is not over, the question continues to be received.

[0099] like Figure 2 As shown, this example also provides a flow chart of the feedback system, including feedback on knowledge points that are "not understood" and feedback on the listening status. We innovatively proposed a judgment formula to judge the "not understood" status.

[0100] Step S9 specifically includes the following steps:

[0101] Step S91: The eye movement trajectory and pupil change characteristics of the eye movement information obtained in step S8 are trained and predicted using a convolutional neural network (CNN) model, and output whether there is any doubt about the knowledge point.

[0102] Step S92: The facial expression information features of the facial information obtained in step S8 are trained and predicted using SVM, and output whether there is any doubt about the knowledge point.

[0103] Step S93: The mouth opening and closing degree feature of the facial information obtained in step S8 is trained and predicted using SVM, and outputs whether there is any doubt about the knowledge point.

[0104] Step S94: Put the characteristic values ​​output from steps S91, S92, and S93 into the following formula to make a final judgment. The formula is as follows:

[0105]

[0106] Where X is a vector feature matrix, each column represents an eigenvalue (a total of 3 eigenvalues), and each row represents a sample, forming an n*3 matrix. For each sample feature, label it (1 for understood / 0 for not understood) and build the matrix Y;

[0107] θ i To assign a weight to each feature value (find the best θ through the training process i value), ∑ i＝1 n(θ1Y i1 +θ2Y i2 +θ3Y i3 ),∑ i＝1 n(Y ji ),∑ j＝1 3(Q j ), P1, P2: If the result reaches the preset threshold (the best threshold is found through the training process) or above, it is considered "understood" and marked as 1; if it does not reach the preset threshold, it is considered "not understood" and marked as 0;

[0108] P1 is the result of judging whether the speaker understands the sample, Q j P is the result of judging whether the audio is understood or not calculated based on the eigenvalues ​​of each column, and P2 is the result of judging whether the audio is understood or not calculated based on each eigenvalue;

[0109] The intersection of the two results is taken, that is, when both are judged as "understood", the result is "understood", and in other cases it is judged as "not understood". If it is judged as not understood, go to step S95.

[0110] Step S95: extract the sound information from the video information, convert it into text, search for relevant knowledge point information in the teaching material database, mark it, and then store it in the database.

[0111] Step S96: Use computer vision technology to estimate the gaze point change feature information of the eye movement information in step S8 and draw a radar chart, and judge the user's concentration by analyzing the stability, duration and position of the gaze point on the screen.

[0112] Step S97: Combine the results of step S95 and step S96, summarize them into a report, and feed back to the student end after class.

[0113] like Figure 3As shown, this embodiment also provides a system for judging the end of speech, which establishes a correlation between emotion, tone and the judgment of the end of speech by extracting effective features of emotion and tone and fusing the two for analysis, and combines lip movement information to make a more accurate judgment, and gives corresponding feedback according to the context.

[0114] Step W6 specifically includes the following steps:

[0115] Step W61: Use the facial expression information, eye movement trajectory, and pupil changes obtained in step W4 to fit the emotion analysis model using SVM, train the model using labeled training data, simulate the target's emotions, and output whether the speech has ended.

[0116] Step W62: Label the lip movements and still frames based on the timestamp of the video data using the lip information obtained in step W4, create a training data set, select a CNN model, and train the labeled data training set to obtain a model that determines whether the lips are moving and whether the speech has ended.

[0117] Step W63: The frequency contour, pitch contour and pitch feature vector of the tone obtained in step W4 are passed through an SVM training model, and the training data with labels are used for training as a basis for judgment.

[0118] Step W64: The results of the model output in steps W61 to W63 are used to make a final judgment using the following formula:

[0119] p final =p 唇动 ∩∑ i∈S w i *p i

[0120] in The set S is {emotion, tone};

[0121] Will Through cost sensitivity analysis, if the preset threshold is reached, it is recorded as 1, if not reached, it is recorded as 0, the performance and cost of the model under different thresholds are estimated, and the optimal threshold is selected; in the above formula:

[0122] p 唇动 Refers to the result obtained by the lip movement information model, the result is 1 (end of speaking) or 0 (not end of speaking), p 情绪 Refers to the result of the sentiment analysis model, the result is 1 or 0, p 音调 The result of the pitch information model is 1 or 0; w 情绪 It refers to the ratio of the F1 score of the sentiment analysis model to the sum of the F1 scores of the sentiment analysis model and the tone information model.音调 The ratio of the F1 score of the tone information model to the sum of the F1 scores of the sentiment analysis model and the tone information model;

[0123] Finally, determine whether the speech is finished based on the formula result. If the speech is finished, go to step W65. If not, continue collecting.

[0124] Step W65: Analyze the text information and contextual emotions obtained in step W4 through a large language model to derive the emotional tendency in the context and understand the expression and intention in the context, give corresponding feedback on the end of the speech, and adjust the other party's emotions. DETAILED DESCRIPTION

[0125] The specific implementation of this embodiment includes the following steps:

[0126] This embodiment includes a MOOC teaching part and a 1v1 teaching part:

[0127] The MOOC teaching part is divided into the student side, the administrator side and the database. The student side users log in, study the MOOC and choose a virtual teacher. The administrator side is responsible for retrieving database resources, returning them to the student side, collecting student information from the student side and storing it in the database, and analyzing user behavior based on the relevant information.

[0128] The 1v1 teaching part is divided into the student side, the administrator side and the database. The student side logs in, selects the 1v1 teaching part, and expresses their doubts. The administrator side is responsible for collecting and analyzing the student side's voice and other information, and returns the question-and-answer video after processing it with the large language model and AI model. It also analyzes user behavior, determines the end of the question, and returns the corresponding conclusion.

[0129] It further includes the database part:

[0130] Create a teacher database, a teaching material database, and a student behavior database. The teacher database includes videos of virtual teachers teaching; the teaching material database includes teacher lecture-related materials and teaching visualization materials to assist teaching; the student behavior database includes collected student-related information.

[0131] Preferably, the MOOC teaching part of the present invention specifically includes the following steps:

[0132] Step S1: Collect the teacher's facial information and voice information when giving a lecture.

[0133] Step S2: Perform data preprocessing on the facial information and voice information in step S1.

[0134] Step S3: Extract features from the facial information in step S2, and extract features from the voice information and convert them into text information:

[0135] For facial information, extract its facial expression information;

[0136] For voice information, its pitch information is extracted.

[0137] Step S4: The feature information in step S3 is stored in the teacher database.

[0138] Step S5: Use programming technology to create teaching visualization videos to assist in understanding knowledge points; collect course-related knowledge point materials and store them in the teaching material database.

[0139] Step S6: The user logs in on the student side, selects the "MOOC teaching" section, selects a specific course and selects the image of the virtual teacher.

[0140] Step S7: After the administrator receives the "MOOC teaching" signal in step S6, it receives the virtual teacher image selected by the user, retrieves the data information of the relevant teachers in the teacher database, and fits it through AIGC to form a virtual teacher teaching video; and retrieves the information of the relevant courses in the teaching resource database, returns the virtual teacher teaching video and the visual teaching video to the student interface, and returns the course-related knowledge point information to the user in PDF format.

[0141] Step S8: After the user starts watching the video returned in step S7, the video information is sliced ​​every ten seconds according to the time, the user's facial information and eye movement information are collected, stored in the student behavior database, and data preprocessing and feature extraction are performed on them:

[0142] For eye movement information, extract the eye movement trajectory, pupil changes and gaze point changes;

[0143] For facial information, extract its facial expression information and the degree of mouth opening and closing.

[0144] Step S9: The user information obtained in step S8 is analyzed through machine learning to obtain a model for evaluating whether the user understands and a model for evaluating the user's concentration. The parts that are judged not to be understood are marked on the knowledge point information in the teaching material database, and a class report is generated based on the concentration and understanding ratio. The class report is stored in the student behavior database and fed back to the user at the end of the class.

[0145] Preferably, the 1v1 teaching part of the present invention includes the following steps:

[0146] Step W1: The user logs in on the student side, selects the “1v1 teaching” section, and chooses the image of the virtual teacher.

[0147] Step W2: After receiving the "1v1 teaching" signal in step W1, the administrator side accepts the virtual teacher image selected by the user, slices the video every ten seconds according to time, and transmits it to the administrator side, and starts receiving the user's voice information, facial information and eye movement information.

[0148] Step W3: Perform data preprocessing on the voice information, facial information and eye movement information obtained in step W2, and convert the voice information into text information.

[0149] Step W4: Perform feature extraction on the voice information, facial information, eye movement information, and text information processed in step W3:

[0150] For facial information, extract lip movement information, facial expression information and mouth opening degree;

[0151] For eye movement information, extract the eye movement trajectory and pupil changes;

[0152] For pitch information, extract its frequency contour, pitch contour and pitch feature vector;

[0153] Extract contextual sentiment from text information;

[0154] After completion, proceed to step W6.

[0155] Step W5: The text information obtained in S3 is transmitted to the large language model, and the corresponding answer is returned. It is put into the AI ​​model together with the image of the virtual teacher obtained in step S2 to generate a virtual teacher video, which is returned to the student end.

[0156] Step W6: The data extracted from features in W4 is put into machine learning to form a model for determining the end of the question. If the conversation is judged to be over, the corresponding ending statement is returned; if the conversation is not over, the question continues to be received.

[0157] Furthermore, step S9 specifically includes the following steps:

[0158] Step S91: The eye movement trajectory and pupil change characteristics of the eye movement information obtained in step S8 are trained and predicted using a convolutional neural network (CNN) model, and output whether there is any doubt about the knowledge point.

[0159] Step S92: The facial expression information features of the facial information obtained in step S8 are trained and predicted using SVM, and output whether there is any doubt about the knowledge point.

[0160] Step S93: The mouth opening and closing degree feature of the facial information obtained in step S8 is trained and predicted using SVM, and outputs whether there is any doubt about the knowledge point.

[0161] Step S94: Put the characteristic values ​​output from steps S91, S92, and S93 into the following formula to make a final judgment. The formula is as follows:

[0162]

[0163] Where X is a vector feature matrix, each column represents an eigenvalue (a total of 3 eigenvalues), and each row represents a sample, forming an n*3 matrix. For each sample feature, label it (1 for understood / 0 for not understood) and build the matrix Y;

[0164] θ i To assign a weight to each feature value (find the best θ through the training process i value), ∑ i＝1 n(θ1Y i1 +θ2Y i2 +θ3Y i3 ),∑ i＝1 n(Y ji ),∑ j＝1 3(Q j ), P1, P2: If the result reaches the preset threshold (the best threshold is found through the training process) or above, it is considered "understood" and marked as 1; if it does not reach the preset threshold, it is considered "not understood" and marked as 0;

[0165] P1 is the result of judging whether the speaker understands the sample, Q j P is the result of judging whether the audio is understood or not calculated based on the eigenvalues ​​of each column, and P2 is the result of judging whether the audio is understood or not calculated based on each eigenvalue;

[0166] The intersection of the two results is taken, that is, when both are judged as "understood", the result is "understood", and in other cases it is judged as "not understood". If it is judged as not understood, go to step S95.

[0167] Step S95: extract the sound information from the video information, convert it into text, search for relevant knowledge point information in the teaching material database, mark it, and then store it in the database.

[0168] Step S96: Use computer vision technology to estimate the gaze point change feature information of the eye movement information in step S8 and draw a radar chart, and judge the user's concentration by analyzing the stability, duration and position of the gaze point on the screen.

[0169] Step S97: Combine the results of step S95 and step S96, summarize them into a report, and feed back to the student end after class.

[0170] Furthermore, step W6 is specifically as follows:

[0171] Step W61: Use the facial expression information, eye movement trajectory, and pupil changes obtained in step W4 to fit the emotion analysis model using SVM, train the model using labeled training data, simulate the target's emotions, and output whether the speech has ended.

[0172] Step W62: Label the lip movements and still frames based on the timestamp of the video data using the lip information obtained in step W4, create a training data set, select a CNN model, and train the labeled data training set to obtain a model that determines whether the lips are moving and whether the speech has ended.

[0173] Step W63: The frequency contour, pitch contour and pitch feature vector of the tone obtained in step W4 are passed through an SVM training model, and the training data with labels are used for training as a basis for judgment.

[0174] Step W64: The results of the model output in steps W61 to W63 are used to make a final judgment using the following formula:

[0175] p final =p 唇动 ∩∑ i∈S w i *p i

[0176] in The set S is {emotion, tone};

[0177] Will Through cost sensitivity analysis, if the preset threshold is reached, it is recorded as 1, if not reached, it is recorded as 0, the performance and cost of the model under different thresholds are estimated, and the optimal threshold is selected; in the above formula:

[0178] p 唇动 Refers to the result obtained by the lip movement information model, the result is 1 (end of speaking) or 0 (not end of speaking), p 情绪 Refers to the result of the sentiment analysis model, the result is 1 or 0, p 音调 The result of the pitch information model is 1 or 0; w 情绪 It refers to the ratio of the F1 score of the sentiment analysis model to the sum of the F1 scores of the sentiment analysis model and the tone information model. 音调 The ratio of the F1 score of the tone information model to the sum of the F1 scores of the sentiment analysis model and the tone information model;

[0179] Finally, determine whether the speech is finished based on the formula result. If the speech is finished, go to step W65. If not, continue collecting.

[0180] Step W65: Analyze the text information and contextual emotions obtained in step W4 through a large language model to derive the emotional tendency in the context and understand the expression and intention in the context, give corresponding feedback on the end of the speech, and adjust the other party's emotions.

Claims

1. A teaching system based on AIGC classroom behavior feedback, characterized in that: The teaching interface is divided into two parts, one part is a virtual teacher, which is a cartoon character or a social celebrity, and the other part is a visual teaching interface for displaying teaching content to help users better understand the teaching content; The AIGC classroom behavior feedback teaching system includes: a MOOC teaching part and a 1v1 teaching part; The MOOC teaching part is divided into a student side, an administrator side and a database. The student side user logs in to conduct MOOC learning and select a virtual teacher; the administrator side is responsible for retrieving database resources, returning them to the student side, and collecting student information from the student side and storing it in the database, and analyzing user behavior based on relevant information; the 1v1 teaching part is divided into a student side, an administrator side and a database. The student side user logs in and selects the 1v1 teaching part to tell the doubts; the administrator side is used to collect and analyze the voice information of the student side, and returns the question-answering video after processing by the large language model and the AI ​​model. It is also used to analyze the user behavior of the student side, determine that the question is over, and return the corresponding ending remarks; The MOOC teaching part includes the following steps: Step S1: Collecting facial information and voice information of the teacher while giving a lecture; Step S2: performing data preprocessing on the facial information and voice information in step S1; Step S3: extracting features from the facial information obtained after the data preprocessing in step S2, extracting features from the voice information obtained after the data preprocessing in step S2, and converting the voice information after the data preprocessing into text information: wherein, extracting facial expression information from the facial information obtained after the data preprocessing, and extracting tone information from the voice information obtained after the data preprocessing; Step S4: storing the characteristic information in step S3 into the teacher database; Step S5: Use programming technology to create a teaching visualization video to assist in understanding knowledge points; collect course-related knowledge point materials and store them in a teaching material database; Step S6: The user logs in on the student terminal, selects the "MOOC teaching" section, selects a specific course and selects the image of the virtual teacher; Step S7: After receiving the "MOOC teaching" signal in step S6, the administrator receives the virtual teacher image selected by the user, retrieves the data information of the relevant teachers in the teacher database, and performs fitting through AIGC to form a virtual teacher teaching video; retrieves the information of the relevant courses in the teaching resource database, returns the virtual teacher teaching video and the visual teaching video to the student interface, and returns the course-related knowledge point information to the user in PDF format; Step S8: After the user starts watching the video returned in step S7, the video information is sliced ​​every ten seconds according to the time, and the user's facial information and eye movement information are collected, stored in the student behavior database, and data preprocessing and feature extraction are performed: for the eye movement information, the eye movement trajectory, pupil changes and gaze point changes are extracted; for the facial information, the facial expression information and the degree of mouth opening and closing are extracted; Step S9: Analyze the user information obtained in step S8 through machine learning to obtain a model for evaluating whether the user understands and a model for evaluating the user's concentration. Mark the parts that are judged not to be understood on the knowledge point data in the teaching material database, generate a class report based on the concentration and the understanding ratio, store the class report in the student behavior database, and provide feedback to the user at the end of the class; Wherein, the step S9 specifically includes the following steps: Step S91: training and predicting the eye movement trajectory and pupil change characteristics of the eye movement information obtained in step S8 using a convolutional neural network (CNN) model, and outputting whether there is any doubt about the knowledge point; Step S92: training and predicting the facial expression information features of the facial information obtained in step S8 using SVM, and outputting whether there is doubt about the knowledge point; Step S93: training and predicting the mouth opening and closing degree feature of the facial information obtained in step S8 using SVM, and outputting whether there is doubt about the knowledge point; Step S94: Based on the eye movement trajectory and pupil change characteristics of the eye movement information output by step S91, the facial expression information characteristics of the facial information output by step S92, and the mouth opening and closing degree characteristics of the facial information output by step S93, a vector feature matrix is ​​constructed, wherein the vector feature matrix is ​​an n*3 matrix, and the vector feature matrix includes n samples, and each sample includes the eigenvalues ​​output by steps S91, S92 and S93 respectively; labeling is performed on each eigenvalue in the vector feature matrix to obtain a matrix Y, wherein the matrix Y is an n*3 matrix, and each eigenvalue corresponds to a label, and the label includes understanding or not understanding. If the label corresponding to the eigenvalue is understanding, the element value of the eigenvalue in the matrix Y is recorded as 1, and if the label corresponding to the eigenvalue is not understanding, the element value of the eigenvalue in the matrix Y is recorded as 0; the matrix Y is put into the following formula for judgment to determine whether the user understands; P=P1∩P2 Wherein, P1 is the result of judging whether the user understands the audio by taking n samples in the matrix Y as a unit, θ i is a weight assigned to each feature value, and the weight is the optimal value determined by the training process, Y i1 is the i-th element in the first column of the matrix Y, i2 is the i-th element in the second column of the matrix Y, i3 is the i-th element in the third column of the matrix Y, and n is the number of rows of the matrix Y; Q j is the result of judging whether the user understands the audio by taking the jth column in the matrix Y as the unit, Y ji is the i-th element in the j-th column of the matrix Y; P2 is the result of judging whether the user understands the audio in units of each eigenvalue in the j-th column of the matrix Y; If any one of the values ​​of P1 and P2 reaches a preset threshold, it is set to 1, indicating that the user understands; if the value does not reach the preset threshold, it is set to 0, indicating that the user does not understand; the intersection of the result value of P1 and the result value of P2 is taken, if the intersection is 1, it is determined that the user understands; if the intersection is not 1, it is determined that the user does not understand, and if it is determined that the user does not understand, the following step S95 is executed; Step S95: extracting sound information from the video information, converting it into text, searching for relevant knowledge point information in the teaching material database, marking it, and then storing it in the database; Step S96: using computer vision technology to estimate the gaze point change feature information of the eye movement information in step S8 and draw a radar chart, and judging the user's concentration by analyzing the stability, duration and position of the gaze point on the screen; Step S97: Combine the results in step S95 and step S96, summarize them into a report, and feed back to the student after class.

2. The AIGC classroom behavior feedback teaching system according to claim 1 is characterized in that: It also includes creating the database: Create a teacher database, a teaching material database, and a student behavior database. The teacher database includes videos of virtual teachers teaching; the teaching material database includes teacher lecture-related materials and teaching visualization materials to assist teaching; the student behavior database includes collected student-related information.

3. The AIGC classroom behavior feedback teaching system according to claim 1 or 2, characterized in that: The 1v1 teaching part includes the following steps: Step W1: The user logs in on the student side, selects the "1v1 teaching" section, and chooses the image of the virtual teacher; Step W2: After receiving the "1v1 teaching" signal in step W1, the administrator accepts the virtual teacher image selected by the user, slices the video every ten seconds according to the time, transmits it to the administrator, and starts receiving the user's voice information, facial information and eye movement information; Step W3: performing data preprocessing on the voice information, facial information and eye movement information obtained in step W2, and converting the voice information into text information; Step W4: extract features from the voice information, facial information, eye movement information and text information processed in step W3: extract lip movement information, facial expression information and mouth opening degree from facial information; extract eye movement trajectory and pupil change from eye movement information; extract frequency contour, pitch contour and pitch feature vector from pitch information; extract context emotion from text information; after completion, proceed to step W6; Step W5: The text information obtained in step W3 is transmitted to the large language model, and the corresponding answer is returned. The answer and the image of the virtual teacher obtained in step W2 are put into the AI ​​model to generate a virtual teacher video, which is returned to the student end; Step W6: Put the data whose features are extracted in W4 into machine learning to form a model for judging the end of the question. If the conversation is judged to be over, the corresponding ending words are returned. If the conversation is not over, continue to receive.

4. The AIGC classroom behavior feedback teaching system according to claim 3 is characterized in that: The step W6 specifically includes the following steps: Step W61: Use the facial expression information, eye movement trajectory, and pupil change obtained in step W4 to fit the emotion analysis model using SVM, use the labeled training data to train the model, simulate the emotion of the recognition target, and output whether the speech is finished; Step W62: label the lip movements and still frames according to the timestamp of the video data using the lip information obtained in step W4, create a training data set, select a CNN model, train the labeled data training set, and obtain a model for determining whether the lips are moving and whether the speech is finished; Step W63: The frequency contour, pitch contour and pitch feature vector of the tone obtained in step W4 are trained through an SVM training model using labeled training data as a basis for judgment; Step W64: Make a final judgment on the results output by the model in steps W61 to W63 using the following formula: p final =p 唇动 ∩∑ i∈S w i *p i in The set S is {emotion, tone}; Will Through cost sensitivity analysis, if the preset threshold is reached, it is recorded as 1, if not reached, it is recorded as 0, the performance and cost of the model under different thresholds are estimated, and the optimal threshold is selected; in the above formula: p 唇动 Refers to the result obtained by the lip movement information model. A result of 1 indicates that the speech has ended, and a result of 0 indicates that the speech has not ended. 情绪 Refers to the result of the sentiment analysis model, the result is 1 or 0, p 音调 The result of the pitch information model is 1 or 0; w 情绪 It refers to the ratio of the F1 score of the sentiment analysis model to the sum of the F1 scores of the sentiment analysis model and the tone information model. 音调 The ratio of the F1 score of the tone information model to the sum of the F1 scores of the emotion analysis model and the tone information model; finally, judging whether the speech is finished according to the formula result, if the speech is finished, proceeding to step W65, if not, continuing to collect; Step W65: Analyze the text information and contextual sentiment obtained in step W4 through a large language model to derive the emotional tendency in the context and understand the expression and intention in the context, give corresponding feedback on the end of the speech, and adjust the other party's emotions.

Images