Teaching two-way evaluation and reexamination method and system based on multiple models of multiple modes

Abstract

The invention discloses a teaching two-way evaluation and reexamination method and system based on multi-modal multiple models, and the method comprises the steps: 100, obtaining multi-modal information related to a specified target, comprising student evaluation teacher information, teacher evaluation student information, student usual score information, student final score information, student active behavior information and student non-inductive behavior information; step 200, adding labels to the multi-modal information and rechecking the multi-modal information; 300, the teacher large model and the student large model are trained, the teacher large model comprises an evaluation teacher large model and an evaluation teacher review large model, and the student large model comprises an evaluation student large model and an evaluation student review large model; and step 400, obtaining a teacher large-scale model prediction result and a student large-scale model prediction result. According to the method, the quality and reliability of intelligent decision making are comprehensively improved, and solid theoretical and practical support is provided for various application scenes.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a teaching two-way evaluation and review method and system based on multi-modal multi-model. Background Art

[0002] Multimodal information fusion refers to the integration of information from different sensors, sources or modalities in order to more comprehensively and deeply understand and analyze a process or phenomenon. In the field of education, multimodal information fusion technology can be applied to the two-way evaluation of the teaching process. By integrating data from different modalities (such as text, audio, video, physiological signals, etc.), it provides more comprehensive and multi-dimensional information, thereby more accurately evaluating teaching effects and learning outcomes.

[0003] However, the current multimodal information fusion technology has the following defects: (1) In the process of multimodal information fusion, since data of different modalities may contain repeated or similar information, there is redundancy in the fused data. This redundant data not only increases the burden of data storage and processing, but also may affect the accuracy of subsequent evaluation and analysis.

[0004] (2) High-level semantic association refers to the deep, nonlinear relationship between different modal data. Existing technologies often have difficulty in effectively capturing these associations, resulting in the fused data being unable to fully and accurately reflect the actual situation of the teaching process.

[0005] (3) In a real-time environment, data from different modalities need to be accurately synchronized for effective analysis and decision-making. However, due to differences in data acquisition methods and processing speeds between different modalities, achieving data synchronization is a technical challenge.

[0006] The current evaluation system uses questionnaires to evaluate students, which results in single evaluation results and low accuracy. In addition, existing two-way interactive technologies often rely on a single interactive method (such as voice or video calls) and lack diversified interactive methods. This limits the communication and interaction between teachers and students, and affects the fluency and interactivity of the teaching process.

[0007] Therefore, it is an urgent problem for technical personnel in this field to propose a teaching two-way evaluation and review method and system based on multimodal and multi-large models to solve the difficulties existing in the prior art. Summary of the invention

[0008] The purpose of the present invention is to provide a teaching two-way evaluation and review method and system based on multimodal and multi-model. Through two-way evaluation, not only the teaching quality of teachers is improved, but also the personalized learning of students is promoted, so that each student can obtain the best learning effect at a suitable rhythm and method.

[0009] To achieve the above object, the present invention provides the following solutions:

[0010] A teaching two-way evaluation and review method based on a multi-modal multi-model includes the following steps:

[0011] Step 100, obtaining multimodal information related to the specified target, including information on students' evaluation of teachers, information on teachers' evaluation of students, information on students' regular grades, information on students' final grades, information on students' active behaviors, and information on students' insensitive behaviors;

[0012] Step 200, labeling and reviewing the multimodal information, wherein the method of labeling and reviewing the teacher-related information in the multimodal information, i.e., step 210, includes:

[0013] Step 211, cleaning and fusing the teacher-related information in the multimodal information to obtain teacher fusion information TMI;

[0014] Step 212: Analyze the TMI obtained in step 211, calculate the average information of the class by class, obtain different suggestions according to the data and grades of different modes, arrange the suggestions in the order of the corresponding data, and obtain the evaluation teacher label ETT;

[0015] Step 213, evaluating whether the ETT obtained in step 212 conforms to the TMI obtained in step 211, scoring the ETT in step 212, and obtaining the RTT;

[0016] Step 300: Train the teacher model and the student model. The teacher model includes the evaluation teacher model and the evaluation teacher review model. The training process of the teacher model, i.e., step 310, includes:

[0017] Step 311: Use the tools provided by the framework to load the pre-trained evaluation teacher model, put the feature value TMI and the label value ETT into the evaluation teacher model for training, and complete the subsequent verification and parameter adjustment steps to obtain ETM;

[0018] Step 312: Use the tools provided by the framework to load the pre-trained evaluation teacher review model, combine TMI and ETT as feature values, RTT as label value, input into the evaluation teacher review model for training, and complete the subsequent verification and parameter adjustment link to obtain RTM;

[0019] Step 400: Use the trained teacher model and student model to conduct a two-way evaluation of the teaching process.

[0020] Preferably, step 100 includes:

[0021] Step 101, obtaining student evaluation teacher information: students evaluate teachers through the teaching business system, fill out questionnaires in the system to score teachers in various dimensions on a 100-point scale, and obtain student evaluation teacher information score_s2t; and enter multimodal evaluation teacher information fusion and multimodal evaluation student information fusion as one of the input information;

[0022] Step 102, obtain the teacher's evaluation student information; in this step, calculate the difference between the student's score for each question and the comprehensive score of all students for each question; obtain the student's evaluation teacher information score_t2s; and enter the multimodal evaluation teacher information fusion and multimodal evaluation student information fusion as one of the input information;

[0023] Step 103, obtaining the student's regular performance information: comprehensively evaluating and grading the student based on the student's class interaction, attendance, homework, etc., to obtain the student's regular performance information SUG;

[0024] Step 104, obtaining the student's final score information: the teacher combines the student's final score and regular score, summarizes them according to the proportion, and obtains the student's final score SFG;

[0025] Steps 105-106, obtain students' active behavior information and unconscious behavior information: collect students' classroom performance through visual acquisition devices bound to people and locations, and convert students' micro-expressions and mental state information into data SUB through information output devices, and input them into multimodal evaluation teacher information fusion and multimodal evaluation student information fusion.

[0026] Preferably, the method of adding labels and reviewing student-related information in multimodal information in step 200, i.e., step 220, includes:

[0027] Step 221, cleaning and fusing the student-related information in the multimodal information to obtain student fusion information SMI;

[0028] Step 222: Analyze the SMI obtained in step 221, give different suggestions to each student according to the data and grades of different modes, arrange the suggestions in the order of the corresponding data, and obtain the evaluation teacher label EST;

[0029] Step 223, evaluate whether the EST obtained in step 222 conforms to the SMI obtained in step 211, score the EST in step 222, and obtain RST.

[0030] Preferably, the student big model in step 300 includes an evaluation student big model and an evaluation student review big model, and the training process of the student big model, i.e., step 320, includes:

[0031] Step 321: Use the tools provided by the framework to load the pre-trained evaluation student model, put the feature value SMI and the label value EST into the evaluation student model for training, and complete the subsequent verification and parameter adjustment steps to obtain ESM;

[0032] Step 322: Use the tools provided by the framework to load the pre-trained student review model, combine SMI and EST as feature values, RST as label values, input them into the student review model for training, and complete the subsequent verification and parameter adjustment steps to obtain RSM.

[0033] Preferably, step 400 includes obtaining the teacher's large model evaluation prediction result, that is, step 410 includes:

[0034] Step 411: Using the TMI obtained in step 211 as a feature value, the teacher evaluation ETR is obtained through the teacher evaluation model trained in step 311;

[0035] Step 412: Review step 411 and generate an evaluation of the ETR obtained in step 411 to ensure that the ETR is as accurate as possible; use the evaluation obtained in step 411 and the feature value obtained in step 211 as input, and use the evaluation teacher review model trained in step 312 to produce an evaluation ETE for each step 411.

[0036] Preferably, step 400 also includes obtaining the student model evaluation prediction result, that is, step 420 includes:

[0037] Step 421: Using the SMI obtained in step 221 as a feature value, the student evaluation ESR is obtained through the large student evaluation model trained in step 321;

[0038] Step 422: Review step 421 and generate an evaluation of the ESR obtained in step 421 to ensure that the ESR is as accurate as possible; use the evaluation obtained in step 421 and the feature value obtained in step 221 as input, and use the evaluation student review model trained in step 322 to generate an evaluation ESE for each step 421.

[0039] Preferably, the learning reminder method is also included: students sit around the n_tableth island table, and a panoramic camera is deployed in the middle of the n_tableth island table. The panoramic camera is used to capture the faces of all students and recognize facial movements. If a student has not entered the learning state for a long time, the camera function of the panoramic camera is started and a prompt sound is issued to remind the student to enter a good learning state.

[0040] The present invention also provides a teaching two-way evaluation and review system based on multimodal multi-large model, which is applied to any of the above-mentioned teaching two-way evaluation and review methods based on multimodal multi-large model, including network equipment, teacher computer, smart home, video acquisition device, information output device, student terminal, multi-functional system server; wherein the multi-functional system server includes teaching business system, large model evaluation system, large model review system, central control system, data fusion system, video analysis system; the network equipment is respectively connected with the teacher computer, smart home, student terminal, video acquisition device, and multi-functional system server;

[0041] The central control system in the multifunctional system server is also connected to the teacher computer and the information output device respectively, and the video analysis system is also connected to the video acquisition device;

[0042] Among them, network equipment is used to provide a network for various devices and servers, facilitating the collection and analysis of video data, the collection of evaluation information, and the transmission of data in various systems and devices;

[0043] Teacher's computer, used to evaluate students, issue classroom tests, and control smart homes through the central control system;

[0044] Smart homes to improve the comfort and safety of learning environments and monitor the environment in classrooms;

[0045] Video acquisition device, used to collect data on students’ active behaviors and students’ involuntary behaviors;

[0046] Information output device, used for teaching assistance, teacher-student interaction and resource sharing;

[0047] Student terminals are used to record students’ evaluation of teachers, students’ regular classroom knowledge point tests, and students’ classroom attendance;

[0048] The multifunctional system server is used to realize all teaching operations, centrally control various equipment in the classroom, and evaluate and review the comprehensive performance of teachers and students.

[0049] Preferably, the video analysis system includes a learning reminder method: students sit around the n_tableth island-shaped table, and a video acquisition device is deployed in the middle of the n_tableth island-shaped table. The video acquisition device is used to capture the faces of all students and recognize facial movements. If a student has not entered the learning state for a long time, the camera function of the video acquisition device is activated and a prompt sound is issued to remind the student to enter a good learning state.

[0050] Preferably, the sign-in method in the student terminal is: bind each student's face to the MAC address of the student terminal during class time, and before get out of class ends, the student signs in through face recognition using the bound MAC address, and each MAC address is bound to a classmate's face.

[0051] Preferably, the method for presenting teaching content in the information output device is: there are n screens on each wall, and 4n screens around the classroom; students use network collaborative editing spreadsheets or collaborative editing document software to publish the presented content on the screen.

[0052] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0053] (1) The present invention adopts a two-way evaluation method in the teaching process, and combines multi-directional cameras to perform multi-dimensional analysis of student status; through two-way evaluation, teachers can obtain students' feedback in real time, help accurately adjust the teaching content and rhythm, thereby improving the pertinence and effectiveness of teaching. Students can also provide feedback based on their own understanding, promoting interaction and optimization in the teaching process.

[0054] (2) This method not only improves the quality of teaching, but also promotes personalized learning, so that each student can achieve the best learning effect at the appropriate rhythm and method. In addition, the multi-directional camera technology captures students' behaviors and states in the classroom from multiple angles, and analyzes students' attention, emotional changes and participation in real time. This intelligent monitoring and analysis can help teachers better understand students' learning status, discover and intervene in potential problems in a timely manner, and optimize teaching strategies. By processing the data captured by the camera through a deep learning model, the accuracy of the assessment of students' status is further improved, ensuring the personalization and adaptability of teaching, and comprehensively improving the effectiveness and intelligence level of classroom teaching. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0056] Figure 1 A flow chart of a teaching two-way evaluation and review method based on a multi-modal multi-model provided by the present invention;

[0057] Figure 2 A structural diagram of a teaching two-way evaluation and review system based on a multi-modal and multi-model provided by the present invention;

[0058] Figure 3This is a location distribution diagram of each device in the system of the present invention in the classroom. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0061] like Figure 1 As shown, the present invention provides a representation learning method based on a multimodal knowledge graph, comprising the following steps:

[0062] (1) Step 100, obtaining multimodal information related to the specified target, including information on students' evaluation of teachers, information on teachers' evaluation of students, information on students' regular grades, information on students' final grades, information on students' active behaviors, and information on students' inactive behaviors;

[0063] Specifically, step 101 includes: obtaining student evaluation teacher information: students evaluate teachers through the teaching business system, fill out questionnaires in the system to score the teachers in various dimensions, and obtain student evaluation teacher information score_s2t (Score from student to teacher); and enter the multimodal evaluation teacher information fusion and multimodal evaluation student information fusion as one of the input information;

[0064] In this step, the questionnaire evaluates teachers based on eight dimensions: satisfaction, student acceptance, teaching speed, teaching content, teaching level, care and interaction, teaching environment, and average difficulty. At the same time, each question is a mandatory question to ensure data integrity. In addition, students can fill in notes, including but not limited to class experience, improvement suggestions, etc.

[0065] The input of step 101 is: the score given by the nth student to the tth teacher for the qth question in the questionnaire in the system is: score_s2t(n, ​​t, q); wherein n represents the number of different students, n=1, 2, ..., N, that is, there are N student numbers in total; q represents the question number of different questions in the questionnaire, q=1, 2, ..., Q, that is, there are Q required questions in total; t represents the teacher number of the evaluation, t=1, 2, ..., T, that is, there are T teachers in total;

[0066] The output of step 101 is result_s2t(t,q), which represents the final score of the qth question of the tth teacher; q=1, 2, ..., Q, that is, there are Q questions in total; t represents the number of the teacher being evaluated, t=1, 2, ..., T, that is, there are T teachers in total;

[0067] The calculation method of step 101 is:

[0068] Sort each question in the student evaluation scores obtained by statistics from high to low, and obtain score_s2t_sorted(n, t, q)n=1, 2, ..., N, that is, there are N student numbers in total; n represents the mandatory question number of different questions in the questionnaire, q=1, 2, ..., Q, that is, there are Q mandatory questions in total; the number of improvements received by the teacher questionnaire is counted as advice(t)t=1, 2, ..., T;

[0069] At the same time, based on a large number of teaching experiences and cases, questionnaires with high scores are often more valuable than those with low scores. On the one hand, high scores are an affirmation of teachers' personal abilities and efforts, which helps teachers get more opportunities in their careers, such as promotion and participation in high-level teaching seminars. Low scores may make teachers feel frustrated and even affect their teaching enthusiasm and career development prospects. On the other hand, high scores provide positive teaching feedback to school management, which helps schools understand which teachers have good teaching results, so as to provide more support in terms of resource allocation, reward mechanisms, etc.

[0070] Therefore, the top part of the ranking is removed in the calculation, and the amount of data is recorded as N top ; and after ranking α·N top The amount of data; where 0<a<1; on this basis, set new weights; that is, the weight of high scores is large, and the weight of low scores is small; set the minimum weight to least_w, and the weights are set to score_s2t_w(n,q), n=1,2,...,N q=1,2,...,Q

[0071] The weight is calculated as:

[0072]

[0073] Calculate the weighted score result_s2t(t,q) of the teacher for this question:

[0074]

[0075] Step 102: Obtain the teacher's evaluation information of the student; in this step, calculate the difference between the score of each question of the student and the comprehensive score of each question of all classmates; obtain the teacher's evaluation information score_t2s (Score iromteacher to student); and enter the multimodal evaluation teacher information fusion and multimodal evaluation student information fusion as one of the input information;

[0076] The input of this step is the same as that of step 101;

[0077] The output of step 102 is result_t2s(n,q), which represents the ranking of the qth question of the nth student;

[0078] In the above step 101, the questionnaire scores are sorted, and the ranking of each student's questionnaire score is recorded as t2S_sorted(n, q); then the calculation method of result_t2s(n, q) is:

[0079]

[0080] Step 103: Obtaining students' usual grades information: comprehensively evaluating and grading students based on their class interaction, attendance, homework, etc., to obtain students' usual grades information SUG (Student's Usual Grades);

[0081] Through the teaching business system, teachers conduct regular tests and midterm exams;

[0082] The input of step 103 is: the scores of the daily tests and mid-term examinations of the students using the teaching business system; score_daily(n, j) represents the score of the jth daily test of the nth student; score_mid(n) represents the mid-term examination score of the nth student; score_home(n, m) represents the mth homework score of the nth student; there are M classes in this semester; n=1, 2, ..., N, that is, there are N student numbers in total, j=1, 2, ..., J, that is, there are J daily tests in total, m=1, 2, ..., M, that is, there are M classes in total;

[0083] The output of step 103 is: the regular score of the student res_score_regular(n), which is the regular score of the nth student; the sign-in score of the student res_sign(n), which is the sign-in score of the nth student;

[0084] To calculate the students' regular grades, teachers will set the weights of the regular homework in the teaching business system. Set teacher_w1 as the regular test weight, teacher_w2 as the midterm exam weight, and teacher_w3 as the homework weight. Teacher_w is defined by the teacher based on subjectivity and experience. Count the students' regular homework submission status and get the regular homework submission rate homework_submit(n); count the students' final homework status and get the final homework submission rate end_term_submit(n);

[0085] At the same time, the Education Bureau sets weights for some high-level evaluation courses and public compulsory courses (such as advanced mathematics, linear algebra, etc.) based on the regular grades and the school's subject rating results, including: regular homework expert weight expert_w1, midterm exam expert weight expert_w2, homework expert weight expert_w3; combining the two weights, we get the i-th comprehensive index w_com i , which is calculated as follows:

[0086] In order to combine these two weights, an adjustment parameter λ (0≤λ≤1) is introduced to balance the relative importance between expert weight and custom weight. When λ=1, it completely relies on expert weight; when λ=0, it completely relies on custom weight; and when 0<λ<1, it is a weighted combination of the two; is the average of the nth student’s regular test scores, is the average of the homework scores of the nth student;

[0087] w_com i =λ.expert_w i +(1-λ).teacher_w i Where 0<λ<1

[0088]

[0089] λ_home.min(level_home, home_zero(n)) is the penalty term, λ_home is the penalty factor. In order to ensure the quality of students' learning and promote a correct learning attitude, level_home is set as the minimum number of times students fail to submit homework on time, and home_zero(n) is the number of times students fail to submit homework on time. Once the number of times a student fails to submit homework on time exceeds this bottom line, his / her regular grades will be reduced as a punishment, so as to encourage students to attach importance to homework submission and abide by learning discipline.

[0090] res_score_regular(n)=score_regular(n)+λ_home.min(level_home, home_zero(n))

[0091] The attendance of students in class is counted in the classroom through the person and location binding device, including:

[0092] During class, each n_group of students sit in a circle, and each student is equipped with a student terminal (such as a tablet) as a person and location binding device; before class, a face scan is performed, and a successful face scan is considered as a successful sign-in for class. At the end of the semester, the number of sign-ins for the student is counted class_sign(n);

[0093] Set the basic sign-in score score_sign;

[0094] λ_sign.min(level_sign, class_sign(n)) is the penalty term, λ_sign is the penalty factor. In order to ensure the quality of students' learning and promote a correct learning attitude, level_sign is set as the minimum tolerable number of sign-ins, and the number of students signing in on time is class_sign(n). Once the number of absences exceeds this bottom line, their regular grades will be reduced as a punishment, so as to encourage students to attend classes on time and abide by learning discipline.

[0095] res_sign(n)=score_sign+λ_sign.min(level_sign, class_sign(n))

[0096] Step 104: Obtain the student's final grade information. The teacher combines the student's final grade and regular grade according to the proportion to obtain the student's final grade SFG (Student's Final Grade);

[0097] The input of step 104 is that the student takes the final exam in class and enters the score into the teaching business system, which is set as score_end_term(n); the student's last final major assignment score is set as score_work(n); and the final major assignment is defended at the same time, which is set as score_answer_work(n); in this class, the knowledge points in this class are tested and defended, and the score is set as score_answer_class(n); n = 1, 2, ..., N; for the convenience of expression, let the variable X in the above step be i _end(n), i=1, 2, 3, 4;

[0098] The output of step 104 is: the student's final score result_end_term(n), and the overall score is result_final(n);

[0099] The teaching business system will store and read the final score h is_end_term(n) calculated by the teacher using the final homework score_h is_work(n), the class defense score score_h is_answer_class(n), the final defense score score_h is_answer_work(n), and the final examination score score_h is_end_term(n);

[0100] Use these historical data scores to calculate weights and get the final score result_end_term(n);

[0101] For the convenience of expression, let the above four historical grade variables, namely, final homework score_his_work(n), class defense score score_his_answer_class(n), final defense score score_his_answer_work(n), and final exam score score_his_end_term(n), be X i _his, i=1, 2, 3, 4; represents the above four variables; let the final score his_end_term(n) be Y_his(n)n=1, 2, ..., N;

[0102] By calculating the Pearson correlation coefficient between each variable and the final grade i _end_term, i = 1, 2, 3, 4, to calculate each corresponding variable weight value end_SFG_w i i=1,2,3,4:

[0103]

[0104] The correlation coefficient ranges from -1 to 1. Since the calculated variable is a positive variable, the value range is 0 to 1. Normalize and calculate the weight:

[0105]

[0106] Calculating Final Grades:

[0107]

[0108] Calculate the overall score, use the final score and the regular score, and set the λ adjustment parameter to adjust the weight of the two. This parameter is set by the teacher:

[0109] result_final(n)=λresult_end_term(n)+(1-λ).res_score_regular(n)0<λ<1

[0110] Step 105: Obtain student active behavior information: In the classroom, configure n_cam visual acquisition devices equipped with high-definition cameras; n_cam=8 is the total number of cameras;

[0111] Based on a lot of experimental experience, it is concluded that a classroom equipped with 8 devices can fully capture the expressions of students in the classroom. If there are more than 8 devices, not only will the cost increase, but also a large number of repeated image samples will be generated;

[0112] The system processes the photo and converts it into a feature vector. Suppose the feature vector of the above check-in photo face_vector_each_student is:

[0113] face_vector_each_student=(v_std1, v_std2...v_std J )

[0114] Assume that the feature vector of the above-mentioned picture captured by the camera face_vector_from_cam is specifically:

[0115] face_vector_from_cam=(v_cam1, v_cam2...v_cam J )

[0116] For the feature vector in the image to be detected and the feature vector in the face sign-in image library during class, calculate the Euclidean distance between them; distance_n_m is represented by the Euclidean distance between the feature vector of the mth image and the feature vector of the nth sign-in face;

[0117]

[0118] Perform matching verification and set the threshold T_distance; if the calculated Euclidean distance distance_n_m < T_distance, the face match is considered successful; because it has been verified in multiple classroom experiments that the number of photos with successful face matching is often less than the actual number of people who sign in for class, the threshold T_distance needs to be appropriately increased to ensure that all the signed-in faces and the collected picture faces are successfully matched;

[0119] After matching, the visual acquisition device extracts human posture recognition in the classroom, extracts the student's standing, sitting, hand-raising and other posture information; at the end of the semester, this information is counted and recorded as the total number of teacher-student interactions act_num(n)n=1, 2, 3...N;

[0120] In class, teachers ask questions and interact online through the teaching business system; students answer questions using configured student terminals (such as tablet computers), and the total number of words in these answers is counted at the end of the semester as the total number of words in teacher-student interaction act_word(n)n=1, 2, 3...N;

[0121] Step 106: Obtaining student unconscious behavior information: In order to collect students' unconscious behaviors in class, including but not limited to learning motivation, class status, etc.; collect students' classroom performance through human and position binding devices and visual collection devices, and convert students' micro-expressions, mental states and other information into data through information output devices, and input them into multi-modal evaluation teacher information fusion and multi-modal evaluation student information fusion SUB (Student Unconscious Behavior);

[0122] The student terminal for signing in during class will ask students to sign in during class; when the nth signed-in student in class fails to sign in during this face recognition, it is regarded as a student's insensitive behavior, that is, the student may lie down in class for some reason and not actively listen to the lecture, which is regarded as a negative insensitive signal; the statistics of this behavior of the nth student in this semester are recorded as face_num(n);

[0123] In class, tasks that require note-taking will be issued. The student terminal will monitor the student's behavior. If the student takes notes as required, it will be considered a good signal. The number of such behaviors of the nth student in the semester is recorded as write_num(n);

[0124] During class, the student terminals (such as tablet computers) in the classroom scan and recognize students' facial expressions from time to time; the specific method is as follows:

[0125] The device takes a picture of the student's face, extracts the picture and records it as emotion_pic(n), and recognizes the student's expression as positive or negative after calculation;

[0126] The expressions of students who actively listen to the teacher and interact in class are positive expressions; the expressions of students who are not interested in what the teacher is saying and are in a low state in class are negative expressions.

[0127] At the end of the semester, statistics are collected to record the number of negative expressions as negative_num(n) and the number of positive expressions as positive_num(n);

[0128] The specific calculation method is as follows:

[0129] The video acquisition device is enhanced based on the ResNet18 neural network architecture, and uses its carefully designed 18-layer convolutional blocks to extract deep features. When the photo data passes through this series of convolutional layers, it is gradually converted into high-level abstract features, and finally output as the predicted probabilities of two expression categories after the last pooling layer. In order to optimize the flow of information and alleviate the gradient vanishing problem in deep networks, ResNet18 cleverly embeds a residual block between every two standard convolutional blocks after the first convolutional block, ensuring that the model can maintain or improve computing power as the depth increases.

[0130] In order to further improve the model's ability to capture key expression features, we introduced an attention mechanism based on ResNet18; these attention modules are carefully deployed at key positions in ResNet18, specifically: before the first convolutional layer, before the second convolutional block, before the sixth convolutional block, before the tenth convolutional block, before the fourteenth convolutional block, and before the final pooling layer. The selection of these positions is intended to allow the model to focus on the most critical information for expression recognition at different stages of feature extraction, thereby enhancing the recognition accuracy and robustness of the model;

[0131] The convolution block includes convolution layer, normalization processing, and activation function;

[0132] The convolutional layer is calculated as follows:

[0133] For the convenience of calculation, N is the number of images in a batch of input, that is, the batch size. The number of channels in each image is C, the length is W, and the width is H. R is the number of output feature maps, that is, the number of output channels.

[0134] For ease of calculation, emotion_pic(n) is set to X, then in a batch, the nth input is X n , the value of the i-th row and j-th column in the c-th channel is x i，j，c，n ;

[0135] Define Y as the output, then in a batch, the value of the rth channel, ith row, jth column in the output corresponding to the nth image is y i，j，r，n ;

[0136] Define p as the padding size, s as the convolution step size, and k as the convolution kernel size. Then the convolution kernel can be expressed as ω k×k , for the convenience of representation, the convolution kernel size is omitted and expressed as ω. For each input image X n, the number of channels of the convolution kernel is the number of channels of the input image, also recorded as C. The number of convolution kernels is the number of feature maps of the output, also recorded as R. Then on the convolution kernel corresponding to the nth image, the value of the i-th row and j-th column on the c-th channel of the r-th convolution kernel is recorded as ω i，j，c，r，n The bias corresponding to each convolution kernel is denoted as b r，n

[0137] Then there is the formula:

[0138]

[0139] The above definition of y i，j，r，n is the value of the rth channel, row i, column j in the output corresponding to the nth image in a batch. Then in a batch, the nth input image X n The value of the rth channel in the corresponding output is recorded as Y r，n . Then in a batch, the nth input image X n The corresponding output is Y n .

[0140] in Y n =[Y1, Y2, ..., Y n ]

[0141] Define the above process, by X n Change to Y n The mapping is f(x), that is, f(X n ; w, b) = Y n

[0142] The normalization calculation method is as follows:

[0143] Define I as the length of the feature map, and J as the width of the feature map. It can be obtained from the definition in the following formula:

[0144]

[0145]

[0146] Defined in a batch, the output value y in each channel r is i，j，r，n The average value is denoted as μ r , the variance is recorded as Then there is a calculation formula:

[0147]

[0148]

[0149] After that, in a batch, each output value of the corresponding channel is normalized, and then scaled and translated, where the scaling factor of the rth channel is denoted by γ r , the translation coefficient of the rth channel is denoted as β r ∈ is a very small constant to prevent the denominator from approaching 0. is the normalized value. In summary, the following formula is obtained:

[0150]

[0151] The activation function is as follows:

[0152] The activation function is ReLU, and each value on each channel of each image in a batch passes through the activation function, and its formula is:

[0153] ReLU(x)=max(0,x)

[0154] ReLU(y i，j，r，n )=max(0,y i，j，r，n )

[0155] To facilitate subsequent calculations, the normalization process and activation function process are omitted, and the convolution layer, normalization processing, and activation function are uniformly recorded as a convolution block. The formula is:

[0156]

[0157] Where k is the size of the convolution kernel. To facilitate the distinction of the mapping relationship between subsequent different convolution blocks, a time step t is defined, and the above formula can be expressed as the mapping relationship under t time steps.

[0158] The attention mechanism module consists of a multi-head self-attention mechanism layer, layer normalization, a feedforward neural network, and layer normalization; the specific calculation is as follows

[0159] Define the input image as X [C，H，W] , where C is the number of channels of the input image, H, W are the sizes of the input image. P is the size of the cut image patch. Thus, we can get: [C，H，W] , when the specified cutting size is P, it can be cut into N patches in total.

[0160]

[0161] For all patches cut from an image, we use X p Indicates. p Each patch in is flattened and mapped into a vector of dimension D through a learnable linear projection, denoted as Z p .

[0162] Zp =Linear(X p )=X p ·w (c×pxp)×D +b D

[0163] After that, a random classification label Z of size [1, D] is generated class and a positional encoding E of size [N+1, D] position , both are learned as a model parameter. p Concatenate and get Z0, with size [N+1, D]:

[0164] Z0=[Z class ; Z p ]+E position

[0165] Define W Q , W K , W V To randomly generate three learnable weights, namely the query weight matrix, the key weight matrix and the value weight matrix; define h as the number of heads of the multi-head attention mechanism, and for each head, there will be its own set of W Q , W K , W V . Knowing that h is the number of heads, we can get the dimensions of the output vectors after three corresponding matrices:

[0166]

[0167] Project Z0 into different feature spaces to calculate the attention scores, where the dimensions of the three matrices are [D, d Q ],[D,d K ],[D,d V ]:

[0168] Q=Z0·W Q

[0169] K=Z0·W K

[0170] V=Z0·W v

[0171] After projection, the dimensions of the query, key, and value are [N+1, d Q ],[N+1,d K ],[N+1,d V ].

[0172] For each head, we first calculate the Q, K, and V of each head when calculating attention. Then we use the scaled dot product attention formula to calculate and get the output of the i-th head: i :

[0173]

[0174] We weight V as the attention weight, and we get the size of Attention(Q, K, V) to be [N+1, d]. Then we concatenate the outputs of each head together:

[0175] Output=Concat(Output1,…,Output i )

[0176] The size of the Output is [N+1, D]. The concatenated result is transformed through a linear layer to map the final output back to the original embedding dimension.

[0177]

[0178]

[0179] in

[0180] For the convenience of representation, the above series of changes from Z0 to Output define MH, that is:

[0181] Output=MH(X)

[0182] After normalization in the first layer, the output is:

[0183] Out = LayerNorm(X+MH(X))

[0184] Then enter the feedforward neural network:

[0185] MLP Out =Dropout(Linear(GELU(Linear(Out))))

[0186] =Dropout((GELU(Out·w1+b1))·w2+b2)

[0187] =Dropout((GELU(LayerNorm(X+MH(X))·w1+b1))·w2+b2)

[0188] After normalization by the second layer, the output is: (LayerNorm is abbreviated as LN, and the activation function is replaced by σ())

[0189] Final Out =LayerNorm(Out+MLP Out )

[0190] =LN(LN(X+MH(X))+(σ(LN(X+MH(X)·W1+b1))·w2+b2)

[0191] To output the corresponding classification, you need to extract the classification tags embedded in the above steps:

[0192] class=Final_Out[:,0]

[0193] The specific residual network and residual block are calculated as follows:

[0194] What the neural network essentially learns is the mapping relationship H(x) between input and output:

[0195] H(x)=y

[0196] It can be summarized as follows: the neural network fits the mapping H(x) through a series of calculations.

[0197] F(x)=H(x)

[0198] In the residual neural network, residual connection is introduced to add an identity mapping. Generally speaking, the jump connection is realized by adding the output x directly to the output of the convolution layer. Therefore, the above formula becomes:

[0199] F(x)+x=H(x)

[0200] Through transformation, we can get:

[0201] F(x)=H(X)-x

[0202] Combining the above convolution blocks, residual networks and attention mechanisms, we can derive the forward calculation formula:

[0203] Let the input of the i-th residual unit be x i , define x0 as the input, y0 as the output after the first layer of convolution, and the series of convolution operations and nonlinear transformation operations are recorded as f i (x i ), then the output of the 1st, 2nd and 3rd residual units is as follows:

[0204] y1=f1(x1)+x1

[0205]

[0206]

[0207] According to mathematical induction, the expression of the features of any deep unit L can be deduced as follows.

[0208]

[0209] After the image x0 enters ResNet and passes through the first convolution block, the input is divided into two branches, one of which continues to enter the convolution block, and the other branch enters the Vision Transformer to obtain y_vit0:

[0210]

[0211] y_vit0=LN(LN(x0+MH(x0))+(σ(LN(x0+MH(x0))·w 0，1 +b 0，1 ))·w 0，2 +b 0，2 )

[0212] After that, when y0 enters the first residual block, it is divided into two branches, one of which continues to enter the residual block to obtain y1, and the other branch enters the Vision Transformer to obtain y_vit1.

[0213] y_vit1=LN(LN(x1+MH(x1))+(σ(LN(x1+MH(x1))·w 1，1 +b 1，1 ))·w 1，2 +b 1，2 )

[0214] y1 enters the first residual block, where h is the identity mapping:

[0215]

[0216]

[0217] After that, when y2 enters the second residual block, it is also divided into two branches. One branch continues to enter the residual block, and the other branch enters the Vision Transformer without going through the classification layer in ViT to obtain y_vit2. Here, y is represented by i. i is the value of the i-th residual block

[0218] y_vit (i+1) / 2 =LN(LN(x i +MH(x i ))+(σ(LN(x i +MH(x i+1 ))·w1+b1))·w2+b2)

[0219]

[0220]

[0221] From the first residual block to the fourth residual block, a total of 5 Vision Transformer labels will be obtained, and the residual block output value y8. First, concatenate and normalize the obtained y_vit1 to y_vit5:

[0222] Out_ViT=LN(Concat(y_vit1,y_vit2,…,y_vit5))

[0223] Then the residual block output value y9 enters the pooling layer and is flattened and normalized:

[0224] Out_ResNet=LN(AdaptiveAvgPool2d(y8))

[0225] Then concatenate the two outputs above and classify them through a multi-layer perceptron:

[0226] Out_Final=MLP(Concat(Out_ViT, Out_ResNet))

[0227] After concatenation, a Linear and Softmax are performed to complete the classification task:

[0228] Label_P=Softmax(Linear(Out_Final))

[0229] =Softmax(Out_Final·w+b)

[0230] The jth type of expression The probability is:

[0231]

[0232] If Label_P i If the expression is greater than 50%, it is regarded as the jth type of expression and statistics are collected at the end of the term. The number of negative expressions of students is negative_num(n), and the number of positive expressions is postive_num(n).

[0233] (2) The labeling and reviewing process 200 is composed of two processes: the labeling and reviewing process 210 for teachers and the labeling and reviewing process 220 for students.

[0234] The teacher labeling and review process 210 consists of three steps from step 211 to step 213:

[0235] Step 211: Multimodal evaluation teacher information fusion: clean and fuse the data obtained in process 100 to obtain multimodal fusion information TMI (Teacher Multimodal Information);

[0236] Step 212: labeling the evaluation teacher information: analyzing the TMI obtained in step 211, calculating the average information of each class, giving different suggestions according to the data and grades of different modes, arranging the suggestions in the order of the corresponding data, forming a paragraph, and obtaining the evaluation teacher tags ETT (Evaluate Teacher Tags);

[0237] Step 213: Evaluate the teacher information tags and manually review and label them: Evaluate whether the ETT obtained in step 212 conforms to the TMI obtained in step 211, score the ETT in step 212, and obtain RTT (Review Teacher Tags); the specific scoring method is as follows: Use the TF-IDF method to convert each tag in EST and SMI into a word vector; for each ETT tag, calculate the cosine similarity between it and the corresponding TMI tag to obtain a similarity reference value a; if the a value is lower than the preset threshold (for example, 0.6), it indicates that there is a large difference between the predicted label and the actual label, which requires the manual scorer to pay great attention; the scorer evaluates the accuracy of the EST based on the similarity reference value a and the understanding of the label content. The scorer gives a final score b in percentage, and the score b given by the scorer is recorded in the system as RTT;

[0238] The student labeling and review process 220 consists of three steps from step 221 to step 223:

[0239] Step 221: Multimodal evaluation student information fusion: clean and fuse the data obtained in process 100 to obtain multimodal fusion information SMI (Student Multimodal Information);

[0240] Step 222: labeling the evaluation student information: Analyze the SMI obtained in step 221, give different suggestions to each student according to the data and grades of different modes, arrange the suggestions in the order of the corresponding data, form a paragraph, and obtain the evaluation teacher tags EST (Evaluate Student Tags);

[0241] Step 223: Evaluate the student information tags and manually review and label them: Evaluate whether the EST obtained in step 222 conforms to the SMI obtained in step 211, score the ETT in step 212, and obtain RST (Review Student Tags); the specific scoring method is as follows: Use the TF-IDF method to convert each tag in the EST and SMI into a word vector; for each EST tag, calculate the cosine similarity between it and the corresponding SMI tag to obtain a similarity reference value a; if the a value is lower than the preset threshold (for example, 0.6), it indicates that there is a large difference between the predicted label and the actual label, which requires the manual scorer to pay great attention; the scorer evaluates the accuracy of the EST based on the similarity reference value a and the understanding of the label content. The scorer gives a final score b in percentage, and the score b given by the scorer is recorded in the system as RST;

[0242] (3) Evaluation large model training process 300, which consists of two processes: teacher large model training process 310 and student large model training process 320:

[0243] The training process 310 for the teacher model consists of two steps, step 311 to step 312:

[0244] Step 311: Evaluating the teacher model training: Use the tools provided by the framework to load the pre-trained large language model, and innovate on the pre-trained large language model; put the feature value TMI and the label value ETT into the large language model for training, and complete the subsequent verification and parameter adjustment to obtain ETM (Evaluating Teacher Model);

[0245] Step 312: Reviewing Teacher Model training: Use the tools provided by the framework to load the pre-trained large language model, combine TMI and ETT as feature values, RTT as label values, input them into the large language model for training, and complete the subsequent verification and parameter adjustment to obtain RTM (Reviewing Teacher Model);

[0246] The student model training process 320 consists of two steps, step 321 to step 322:

[0247] Step 321: Evaluating the student model training: Use the tools provided by the framework to load the pre-trained large language model and innovate on the pre-trained large language model; put the feature value SMI and the label value EST into the large language model for training, and complete the subsequent verification and parameter adjustment to obtain ESM (Evaluating Student Model);

[0248] Step 322: Evaluate the student review model training: Use the tools provided by the framework to load the pre-trained large language model, combine SMI and EST as feature values, RST as label values, input them into the large language model for training, and complete the subsequent verification and parameter adjustment to obtain RSM (Reviewing Student Model);

[0249] (4) Obtaining model evaluation result process 400, which consists of two processes: teacher large model training process 410 and student large model training process 420:

[0250] The evaluation result process 410 of the teacher model consists of two steps, namely, step 411 to step 412:

[0251] Step 411: Obtain the prediction result of the teacher evaluation model: This step is used as the prediction result of step 311, and the TMI obtained in step 211 is used as the feature value. Through the large language model ETM trained in step 311, a section of teacher evaluation ETR (Evaluate Teacher Results) is obtained;

[0252] Step 412: Obtain the prediction results of the teacher evaluation review model: This step is to review step 411, generate an evaluation of the ETR obtained in step 411, and ensure that the ETR is as accurate as possible; using the evaluation ETR obtained in step 411 and the feature value of the TMI obtained in step 211 as input, through the review large language model RTM trained in step 312, generate the evaluation ETE (Evaluation of Teacher Evaluation) for each step 411;

[0253] The student model evaluation result process 420 consists of two steps, step 421 to step 422:

[0254] Step 421: Obtain the prediction result of the large model for evaluating students: This step is used as the prediction result of step 321, and the SMI obtained in step 221 is used as the feature value. Through the large language model ESM trained in step 321, an evaluation ESR (Evaluate Student Results) about the student is obtained;

[0255] Step 422: Obtain the prediction results of the evaluation student review model: This step is to review step 421, generate an evaluation of the ESR obtained in step 421, and ensure that the ESR is as accurate as possible; using the evaluation ESR obtained in step 421 and the feature value of the SMI obtained in step 221 as input, through the review large language model RSM trained in step 322, generate the evaluation ESE (Evaluation of Student Evaluation) for each step 421;

[0256] A teaching two-way evaluation and review system based on multimodal multi-large model, applied to any of the above-mentioned teaching two-way evaluation and review methods based on multimodal multi-large model, including network equipment 100, teacher computer 200, smart home 400, video acquisition device 500, information output device 600, student terminal 700, multi-function system server 800; wherein the multi-function system server includes teaching business system 801, large model evaluation system 802, large model review system 803, central control system 804, data fusion system 805, video analysis system 806; network equipment 100 is respectively connected to teacher computer 200, smart home 400, student terminal 700, video acquisition device 500, multi-function system server 800;

[0257] The central control system 804 in the multifunctional system server 800 is also connected to the teacher computer 200 and the information output device 600, and the video analysis system 806 is also connected to the video acquisition device 500;

[0258] Among them, the network device 100 is used to provide a network for various devices and servers, facilitating the collection and analysis of video data, the collection of evaluation information, and the transmission of data in various systems and devices;

[0259] The teacher machine 200 is used to implement the teacher's evaluation of students, the issuance of classroom tests, and the control of the smart home through the central control system;

[0260] Smart Home 400, which is used to improve the comfort and safety of the learning environment and monitor the environment in the classroom;

[0261] Video collection device 500, used to collect data on students' active behaviors and students' insensitive behaviors;

[0262] Information output device 600, used for teaching assistance, teacher-student interaction and resource sharing;

[0263] Student terminal 700 is used to record students' evaluation of teachers, students' regular classroom knowledge point tests and students' classroom attendance;

[0264] The multifunctional system server 800 is used to realize all teaching operations, centrally control various equipment in the classroom, and evaluate the comprehensive performance of teachers and students and review the evaluation.

[0265] (1) Network device 100

[0266] The network device 100 is a device used to connect various devices and servers and perform data transmission. It can be a router or a switch. The router model is AR6280-S. First, the network device 100 is connected to the network cable interface in the classroom through a straight-through cable. Secondly, the visual acquisition device 500 is connected to the USB interface of the network device 100 through a network cable. The teacher computer 200, the student terminal device 700, the smart home 400, and the multi-functional system server 700 are all connected to the network device 100 through Ethernet.

[0267] (2) Teacher machine 200

[0268] The teacher computer 200 is a multifunctional device used by teachers for daily teaching, student management, classroom interaction, and resource sharing. It includes three workstations, all of which use Dell computers. As the core equipment of the smart classroom, the teacher computer 200 achieves multifunctional integration by connecting with multiple devices such as the information output device 600, the smart home 400, and the student terminal device 300. First, the teacher computer 200 is connected to the USB interface of the information output device 600 via a data cable; secondly, the teacher computer 200 is connected to the student terminal device 300 and the multifunctional system server 800 via Ethernet;

[0269] (3) Smart Home 400

[0270] Smart home 400 is a home that realizes automated operation through wireless connection and remote control, including smart curtains, smart air conditioners, door closers, magnetic locks, and head-mounted virtual reality devices, with models of DT82TV, RZ-ACT, SW100Slim, DS-K4H258S, and Vision Pro. The network device 100 obtains an IP address through DHCP and communicates with other devices (such as smart curtains and smart air conditioners). Both smart home 400 and the teacher computer 200 are connected to the network device 100 through Ethernet to realize the control of the smart home 400 by the teacher computer 200.

[0271] In the smart home 400, the opening and closing of the classroom door is controlled by powering on and off the host of the central control system;

[0272] There are two situations: (1) The door closer is in the normally closed state. When a person issues a door closing command through the central control system host, the central control system host starts the door closer and demagnetizes the access control. At this time, the door closer pulls the door closed and the access control holds the door so that it cannot be opened. (2) The door closer is set to the normally open state. When a person issues a door opening command through the central control host, the central control system host starts the door closer and demagnetizes the access control. At this time, the door closer pulls the door open.

[0273] The smart home control method in the above device is: deploy all the control buttons of the smart home in the form of a website in the same network as the central control host, use a head-mounted virtual reality device to open the website, and realize the control of the smart home;

[0274] (4) Video acquisition device 500

[0275] The visual acquisition device 500 is a device for collecting classroom video data, including 8 visual acquisition devices: visual acquisition device 501 to visual acquisition device 508, which can be a CCD camera or a 4K high-definition camera. The model of the 4K high-definition camera is UV420SCD.

[0276] The visual acquisition devices 501 to 508 convert the actual scene into a digital image or video through their own image sensors, and convert it into a digital signal for subsequent processing and analysis; the visual acquisition devices 501 to 508 are connected to the communication interface of the network device 100 through their own internal data bus or their own communication interface using a data cable; the network device 100 is connected to the HDMI interface of the video analysis system server 706 through a network cable or a high-definition cable;

[0277] The video acquisition method in the visual acquisition device 501 is as follows: according to a large number of experiments, it is found that at least 8 visual acquisition devices must be deployed to capture most of the faces of people sitting in different directions in the environment; then, a face recognition method is used to remove duplicate faces, so that face videos and images of different people in as many environments as possible can be collected;

[0278] (5) Information output device 600

[0279] The information output device 600 is a device for displaying teaching materials, and includes 12 information output devices: information output device 601 to information output device 612. The information output device 600 may be a liquid crystal display screen, and a seewo 86-inch all-in-one touch screen may be used;

[0280] The information output devices 601 to 612 are devices that convert electrical signals into visible images through the change of optical properties of liquid crystal materials under the action of electric fields; the information output devices 601 to 612 are connected to the HDMI interface of the central control system 804 through HDMI cables, and the central control system 804 is connected to the teacher computer 200 through Ethernet, and the information to be displayed can be transmitted to the liquid crystal display screen through Ethernet;

[0281] The teaching content presentation method in the information output device 600 is: replace one screen with n screens on one wall, each wall has n screens, so that each screen can simultaneously display theoretical teaching, practical teaching, and teaching interaction, and realize non-confrontational presentation of multiple types of content (no need to frequently switch on the same screen);

[0282] The method for presenting teaching content in the above device is: use 4n screens connected together, surrounding the classroom, with n screens on each wall, develop or use network collaborative editing spreadsheets or collaborative editing document software, and ask students to publish the content they want to present on the screen closest to them (one of the 4n screens), so that all students can present their own content, and each student can see the presentation results of all students.

[0283] (6) Student terminal device 700

[0284] The student terminal device 700 is a device for accessing learning resources, real-time interaction between teachers and students, online assessment and class attendance, including student terminal devices 711 to 796, which can be a tablet computer or other smart electronic device, and can use a Huawei tablet computer, model MatePad 2023; the student terminal device 700 is connected to the teacher machine 200 through the network device 100 to realize the interactive management between teachers and students;

[0285] The sign-in method in the student terminal device 700 is as follows: each time a class is held, each student's face is bound to the MAC address of the tablet computer where the student is located. Before the get out of class ends, only this student can sign in through face recognition using the bound MAC address. Each MAC address can only be bound to the face of one classmate, thereby solving the problem of signing in on behalf of others.

[0286] (7) Multi-function system server 800

[0287] The multifunctional system server 800 is a device for acquiring and processing classroom data, including a teaching business system 801, a large model evaluation system 802, a large model review system 803, a central control system 804, a data fusion system 805, and a video analysis system 806; the multifunctional system server 800 is connected to the USB interface of the network device 100 through an industrial line and a data line. The multifunctional system server 800 can be a computer or a programmable system server, and can include two servers: an AIGC backup system server workstation and an AIGC server workstation;

[0288] (7.1) The teaching business system 801 is a comprehensive server for implementing all teaching businesses, and is connected to the video analysis system and the data fusion system through the network device 100;

[0289] (7.2) The large model evaluation system 802 is a server that evaluates the comprehensive performance of teachers and students. The large model evaluation system 802 is connected to the large model review system 803 and the data fusion system 805 through the network device 100;

[0290] (7.3) The large model review system 803 is a server that reviews the evaluations of teachers and students to ensure that the evaluation content is correct. The large model review system 803 is connected to the large model evaluation system 802 through the network device 100;

[0291] (7.4) The central control system 804 is a server that centrally controls various sound, light, and electrical devices in the classroom. The central control system 804 is connected to the smart home 400 through a wireless network and the network device 100 to achieve centralized control of the smart home appliances in the classroom. At the same time, the central control system 804 is connected to the information output device through an industrial line to achieve control of the smart LCD display in the classroom;

[0292] (7.5) The data fusion system 805 is a server that performs fusion calculations on the data in the teaching business system 801 and the video analysis system 806; the data fusion system 805 is connected to the large model review system and the video analysis system through the network device 100;

[0293] (7.6) The video analysis system 806 is a server that analyzes the image or video input by the visual acquisition device 500; first, the video analysis system 806 is connected to the USB interface of the network device 100 via a network cable, and the network device 100 is then connected to the visual acquisition device via a data cable to transmit the real-time data collected by the visual acquisition device 500 to the video analysis system server 806; secondly, the video analysis system 806 is connected to the data fusion system 805 via the network device 100;

[0294] The present invention also discloses a learning reminder method: if students sit around an n_tableth island table, a 4K high-definition camera can be deployed in the middle of the n_tableth island table, and the panoramic camera can capture the faces of all students. Through facial action recognition, if their expressions remain unchanged for a long time or they close their eyes or lie on the table to sleep or eat or drink water, the photo-taking function of the 4K high-definition camera is started and a prompt sound is issued to remind the students to enter a good learning state.

[0295] The present invention provides a teaching two-way evaluation and review method and system based on multimodal and multi-large models, introduces a dual-model review mechanism, and conducts in-depth evaluation of the suggestions generated by the artificial intelligence model to significantly enhance its scientificity and reliability. Through an independent review model, the mechanism can verify the decisions output by the main model in multiple dimensions to ensure the accuracy and consistency of the results. The review model uses advanced data analysis technology to deeply explore the potential deviations in the output of the main model, provide feedback and optimize suggestions in real time, and form a closed-loop correction. This review process not only improves the transparency and trust of model decisions, but also promotes collaborative innovation between different models, making the final output more accurate and in line with actual needs. In addition, relying on the continuous evolution of deep learning algorithms, the mechanism can flexibly adjust in the face of complex situations and maintain efficient review capabilities, thereby comprehensively improving the quality and reliability of intelligent decision-making, and providing solid theoretical and practical support for various application scenarios.

[0296] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0297] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A teaching two-way evaluation and review method based on multi-modal and multi-model, characterized in that: The following steps are involved: Step 100, obtaining multimodal information related to the specified target, including information on students' evaluation of teachers, information on teachers' evaluation of students, information on students' regular grades, information on students' final grades, information on students' active behaviors, and information on students' insensitive behaviors; Step 200, labeling and reviewing the multimodal information, wherein the method of labeling and reviewing the teacher-related information in the multimodal information, i.e., step 210, includes: Step 211, cleaning and fusing the teacher-related information in the multimodal information to obtain teacher fusion information TMI; Step 212: Analyze the TMI obtained in step 211, calculate the average information of the class by class, obtain different suggestions according to the data and grades of different modes, arrange the suggestions in the order of the corresponding data, and obtain the evaluation teacher label ETT; Step 213, evaluating whether the ETT obtained in step 212 conforms to the TMI obtained in step 211, scoring the ETT in step 212, and obtaining the RTT; Step 300: Train the teacher big model and the student big model. The teacher big model includes the evaluation teacher big model and the evaluation teacher review big model. The training process of the teacher big model, i.e., step 310, includes: Step 311: Use the tools provided by the framework to load the pre-trained evaluation teacher model, put the feature value TMI and the label value ETT into the evaluation teacher model for training, and complete the subsequent verification and parameter adjustment steps to obtain ETM; Step 312: Use the tools provided by the framework to load the pre-trained evaluation teacher review model, combine TMI and ETT as feature values, and RTT as label values, input them into the evaluation teacher review model for training, and complete the subsequent verification and parameter adjustment steps to obtain RTM; Step 400: Use the trained teacher model and student model to conduct a two-way evaluation of the teaching process.

2. According to claim 1, a teaching two-way evaluation and review method based on multi-modal multi-model is characterized in that: Step 100 includes: Step 101, obtaining the student evaluation teacher information: students evaluate teachers through the teaching business system, fill out questionnaires in the system to score the teachers in various dimensions, and obtain student evaluation teacher information score_s2t; and enter the multimodal evaluation teacher information fusion and multimodal evaluation student information fusion as one of the input information; Step 102, obtaining the teacher's evaluation student information; in this step, calculating the difference between the student's score for each question and the comprehensive score of all students for each question; obtaining the student's evaluation teacher information score_t2s; and entering the multimodal evaluation teacher information fusion and multimodal evaluation student information fusion as one of the input information; Step 103, obtaining the student's regular performance information: comprehensively evaluating and grading the student based on the student's class interaction, sign-in, homework, etc., to obtain the student's regular performance information SUG; Step 104, obtaining the student's final grade information: the teacher combines the student's final grade and regular grades, summarizes them according to the proportion, and obtains the student's final grade SFG; Steps 105-106, obtaining the active behavior information and unconscious behavior information of the students: collecting students' classroom performance through visual acquisition devices bound to people and locations, and converting students' micro-expressions and mental state information into data SUB through information output devices, and inputting them into multimodal evaluation teacher information fusion and multimodal evaluation student information fusion.

3. According to claim 1, a teaching two-way evaluation and review method based on multi-modal and multi-model is characterized in that: The method of adding labels and reviewing the student-related information in the multimodal information in step 200, i.e., step 220, includes: Step 221, cleaning and fusing the student-related information in the multimodal information to obtain student fusion information SMI; Step 222: Analyze the SMI obtained in step 221, give different suggestions to each student according to the data and grades of different modes, arrange the suggestions in the order of the corresponding data, and obtain the evaluation teacher label EST; Step 223, evaluate whether the EST obtained in step 222 conforms to the SMI obtained in step 211, score the EST in step 222, and obtain RST.

4. According to claim 1, a teaching two-way evaluation and review method based on multi-modal multi-model is characterized in that: In step 300, the student model includes an evaluation student model and an evaluation student review model. The training process of the student model, i.e., step 320, includes: Step 321: Use the tools provided by the framework to load the pre-trained evaluation student model, put the feature value SMI and the label value EST into the evaluation student model for training, and complete the subsequent verification and parameter adjustment steps to obtain ESM; Step 322: Use the tools provided by the framework to load the pre-trained student review model, combine SMI and EST as feature values, RST as label values, input them into the student review model for training, and complete the subsequent verification and parameter adjustment steps to obtain RSM.

5. According to claim 1, a teaching two-way evaluation and review method based on multi-modal and multi-model is characterized in that: Step 400 includes obtaining the teacher's large model evaluation prediction result, that is, step 410 includes: Step 411: using the TMI obtained in step 211 as a feature value, and using the teacher evaluation model trained in step 311, obtain the teacher evaluation ETR; Step 412: Review step 411 and generate an evaluation of the ETR obtained in step 411 to ensure that the ETR is as accurate as possible; use the evaluation obtained in step 411 and the feature value obtained in step 211 as input, and use the evaluation teacher review model trained in step 312 to produce an evaluation ETE for each step 411.

6. According to claim 5, a teaching two-way evaluation and review method based on multi-modal multi-model is characterized in that: Step 400 also includes obtaining the student large model evaluation prediction result, that is, step 420 includes: Step 421: using the SMI obtained in step 221 as a feature value, and using the student evaluation model trained in step 321, obtaining an evaluation ESR for the student; Step 422: Review step 421 and generate an evaluation of the ESR obtained in step 421 to ensure that the ESR is as accurate as possible; use the evaluation obtained in step 421 and the feature value obtained in step 221 as input, and use the evaluation student review model trained in step 322 to generate an evaluation ESE for each step 421.

7. A teaching two-way evaluation and review system based on a multimodal multi-scale model, applied to the teaching two-way evaluation and review method based on a multimodal multi-scale model described in any one of claims 1-6, characterized in that: Including network equipment, teacher computers, smart homes, video acquisition devices, information output devices, student terminals, and multi-functional system servers; The multifunctional system server includes a teaching business system, a large model evaluation system, a large model review system, a central control system, a data fusion system, and a video analysis system; the network device is respectively connected to the teacher's computer, the smart home, the student terminal, the video acquisition device, and the multifunctional system server; The central control system in the multifunctional system server is also connected to the teacher computer and the information output device respectively, and the video analysis system is also connected to the video acquisition device; The network equipment is used to provide a network for various devices and servers, facilitating the collection and analysis of video data, the collection of evaluation information, and the transmission of data in various systems and devices; The teacher machine is used to implement the teacher's evaluation of students, the issuance of classroom tests, and the control of smart homes through the central control system; The smart home is used to improve the comfort and safety of the learning environment and monitor the environment in the classroom; The video acquisition device is used to collect data on students' active behaviors and students' insensitive behaviors; The information output device is used for teaching assistance, teacher-student interaction and resource sharing; The student terminal is used to record students' evaluation of teachers, students' regular classroom knowledge point tests and students' classroom attendance; The multifunctional system server is used to realize all teaching operations, centrally control various equipment in the classroom, and evaluate the comprehensive performance of teachers and students and review the evaluation.

8. The teaching two-way evaluation and review system based on multi-modal and multi-model according to claim 8 is characterized in that: The video analysis system includes a learning reminder method: students sit around the n_tableth island table, and a video acquisition device is deployed in the middle of the n_tableth island table. The video acquisition device is used to capture the faces of all students and recognize facial movements. If a student has not entered the learning state for a long time, the camera function of the video acquisition device is activated and a prompt sound is issued to remind the student to enter a good learning state.

9. The teaching two-way evaluation and review system based on multi-modal and multi-model according to claim 8 is characterized in that: The sign-in method in the student terminal is: bind each student's face to the MAC address of the student terminal during class time. Before get out of class, the student signs in through face recognition using the bound MAC address. Each MAC address is bound to a classmate's face.

10. The teaching two-way evaluation and review system based on multi-modal and multi-model according to claim 8 is characterized in that: The method of presenting teaching content in the information output device is as follows: there are n screens on each wall, and 4n screens around the classroom; students use network collaborative editing spreadsheets or collaborative editing document software to publish the presented content on the screen.

Images