Teacher-student interaction determination method and system based on teaching events and student behavior

Abstract

The application discloses a teacher-student interaction judgment method and system based on teaching events and student behaviors, obtains teaching events, judges the relationship between the teaching events and response time points, and obtains multiple student response time points. A multi-modal student state set of the multiple student response time points is obtained. Based on the multi-modal student state set of the multiple student response time points, the states of teachers and students in a set teaching time length are judged, and student teaching state values are obtained. Under the teaching events, the students have certain reaction times, so a combined state space which is both correlated and distinguishable is constructed according to the reaction times. Therefore, classroom expression behavior recognition, multi-modal learning state analysis of interactive sentences and teaching interaction recognition are performed. The technical effect that the learning degree of students to the teaching events is more accurately judged can be achieved.

Description

Technical Field

[0001] This invention relates to the field of intelligent analysis technology for classroom teaching, and more specifically, to a method and system for determining teacher-student interaction based on teaching events and student behavior. Background Technology

[0002] Students' learning status in the classroom is a latent state that is difficult to observe directly and usually needs to be inferred indirectly based on students' overt behavior and task performance. Available observations include facial movement units, eyelid and eyebrow changes, head posture, fixation area, sitting posture and hand movements, speech and speech prosody, tablet tapping and writing, answer accuracy and response time, as well as optional physiological indicators such as heart rate variability, skin conductance or eye movement.

[0003] Existing classroom status recognition solutions often employ single-frame facial expression classification, single behavior rules, or simultaneous splicing of multiple modalities. They fail to fully utilize the knowledge points the teacher is currently teaching and the fact that students' responses lag behind the teacher's. For example, "looking down" could indicate taking notes, checking a tablet, thinking, fatigue, or being off-task; "frowning" could correspond to either effective deep understanding or cognitive overload. Judging status solely based on the behavior itself, without considering the teaching context, is prone to ambiguity and false alarms. Furthermore, different students exhibit stable individual behavioral habits; judging solely based on a uniform class threshold may misinterpret individual baseline differences as abnormal states.

[0004] Therefore, in summary, a method is needed that uses teaching events as conditions, can explicitly model response delays and students' multimodal states, and can distinguish different student states. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for determining teacher-student interaction based on teaching events and student behavior, in order to solve the problem in the prior art: how to use teaching events as time and semantic anchors to asynchronously fuse classroom videos, student audio, tablet operation logs and available physiological indicators for quality perception, and jointly estimate the student's cognitive load, learning input and response effectiveness for the current knowledge point or teaching behavior under the condition of allowing response time delay and modality loss.

[0006] In a first aspect, embodiments of the present invention provide a method for determining teacher-student interaction based on teaching events and student behavior, including:

[0007] Acquire teaching events; the teaching events include the start time point of the teaching event, the end time point of the teaching event, and the teaching statements;

[0008] Based on the teaching events, the relationship between the teaching events and response time points is determined, and multiple student response time points are obtained.

[0009] Obtain a multimodal student state set at multiple student response time points; the multimodal student state set includes student face images, student body images, and student statements;

[0010] Based on the multimodal student state set at multiple student response time points, the states of teachers and students during a set teaching time are determined to obtain student teaching state values; the student teaching state values ​​represent the degree to which students have learned the teacher's teaching content.

[0011] Optionally, the multimodal student state set based on the multiple student response time points is used to determine the states of teachers and students within a set teaching time length to obtain student teaching state values, including:

[0012] Based on the student body images at the multiple student response time points, a first feature vector and posture time points are obtained; the posture time points are the time points that characterize the student's posture in response to the teaching event; the first feature vector represents the change characteristics of the student's posture at the multiple student response time points.

[0013] The set of multimodal student states corresponding to the attitude time points is taken as the first set of multimodal student states.

[0014] Based on the first multimodal student state set, a second feature vector is obtained; the second feature vector represents the multimodal features of facial expressions, postures, and student statements at the time point of student's response to teaching events.

[0015] Based on the second feature vector and the first feature vector, multimodal feature fusion is performed to obtain the student's teaching status value.

[0016] Optionally, obtaining the first feature vector and pose time points based on the student body images at the multiple student response time points includes:

[0017] Based on the student's body image, joint points are detected to obtain a set of joint point locations; the set of joint point locations includes multiple joint point categories and their corresponding locations.

[0018] Multiple sets of joint point locations are obtained by corresponding to multiple student body images;

[0019] Based on the set of joint positions, a posture feature vector is obtained; multiple posture feature vectors are obtained corresponding to multiple student body images;

[0020] Multiple pose feature vectors are input into the LSTM structure to obtain the first feature vector;

[0021] Based on the set of multiple joint positions, the attitude time points are obtained.

[0022] Optionally, obtaining the attitude time points based on the plurality of joint position sets includes:

[0023] Obtain the largest circle containing the set of joint point positions to obtain the attitude circle;

[0024] Calculate the area of ​​the attitude circle to obtain the attitude area;

[0025] Multiple sets of joint locations correspond to multiple pose areas;

[0026] The student response time point corresponding to the largest posture area among multiple posture areas is taken as the posture time point.

[0027] Optionally, obtaining the attitude feature vector based on the set of joint positions includes:

[0028] The association relationships between multiple joint categories are marked to obtain the joint connectivity matrix;

[0029] Multiple key point locations are input into the first feature extraction network to obtain key point location feature vectors; multiple key point location feature vectors are obtained corresponding to multiple student response time points;

[0030] The joint position feature vectors and joint connectivity matrix are input into a graph convolutional network to obtain attitude feature vectors; multiple joint position feature vectors correspond to multiple attitude feature vectors.

[0031] Optionally, obtaining the second feature vector based on the first multimodal student state set includes:

[0032] The student face images in the first multimodal student state set are input into a face feature extraction network to extract features and obtain a first face feature vector.

[0033] Based on the student body images in the first multimodal student state set, a first body feature vector is obtained through pose detection; the first body feature vector represents the pose of the limbs.

[0034] The first facial feature vector and the first body feature vector are fused to obtain the second feature vector.

[0035] Optionally, the teaching statements and student statements can be input into a large language model for keyword matching to obtain semantic matching values;

[0036] Multiply the semantic association value by the expansion constant and then divide by the length of the student's statement to obtain the student-teacher statement association value.

[0037] Optionally, obtaining the student's teaching status value based on the second feature vector and the first feature vector includes:

[0038] If the correlation value between the student and teacher statements is greater than the correlation threshold, the student statement is input into the statement feature extraction network to obtain the statement feature vector;

[0039] The first feature vector, the second feature vector, and the sentence feature vector are concatenated to obtain the concatenated vector;

[0040] The concatenated vector is input into the discriminant network to obtain the student's teaching status value.

[0041] Optionally, based on the teaching event, the relationship between the teaching event and the response time point is determined to obtain multiple student response time points, including:

[0042] The teaching time period is obtained based on the end time and start time of the teaching event.

[0043] The formula for calculating the teaching time period is as follows:

[0044]

[0045] Among them, the The starting point of the teaching event. The point in time when the teaching event ends. This refers to the timeframe in advance during which students can respond before the end of the instructional statement. This refers to the buffer time that students have to think and react after a teaching event has ended. Indicates a teaching period;

[0046] Multiple time points are evenly extracted from the teaching period to obtain multiple student response time points.

[0047] Secondly, embodiments of the present invention provide a teacher-student interaction determination system based on teaching events and student behavior, including:

[0048] The first acquisition module is used to acquire teaching events; the teaching events include the start time point of the teaching event, the end time point of the teaching event, and teaching statements.

[0049] Based on the teaching events, the relationship between the teaching events and response time points is determined, and multiple student response time points are obtained.

[0050] The second acquisition module is used to acquire a multimodal student state set at multiple student response time points; the multimodal student state set includes student face images, student body images, and student statements;

[0051] The discrimination module is used to discriminate the states of teachers and students within a set teaching time length based on the multimodal student state set at multiple student response time points, and obtain the student teaching state value; the student teaching state value represents the student's learning degree of the teacher's teaching content.

[0052] Compared with the prior art, the embodiments of the present invention achieve the following beneficial effects:

[0053] This invention also provides a method and system for determining teacher-student interaction based on teaching events and student behavior.

[0054] This invention determines a student's learning status in response to a teaching event based on the individual student's state at a single response time point and the overall state changes of the student body across multiple response time points. Since students struggle to understand earlier concepts, they will find it difficult to learn later ones. Therefore, the overall student posture can be used to adjust the student's state, followed by a judgment at a specific time point. The use of posture circles and graph structures in the middle allows for a more accurate assessment of the student's learning status in response to the teaching event.

[0055] In summary, because students have a certain reaction time in instructional events, a joint state space that is both interconnected and distinguishable is constructed based on this reaction time. This space is then integrated with classroom facial expression and behavior recognition, interactive statements, multimodal learning state analysis, and instructional interaction recognition. This approach achieves a more accurate assessment of students' learning levels in relation to instructional events. Attached Figure Description

[0056] Figure 1 This is a flowchart of a teacher-student interaction determination method based on teaching events and student behavior provided in an embodiment of the present invention.

[0057] Figure 2 This is a schematic diagram illustrating the acquisition of posture feature vectors in a teacher-student interaction determination method based on teaching events and student behavior provided in an embodiment of the present invention.

[0058] Figure 3 This is a schematic diagram illustrating the relationship between teaching events and response time points in a teacher-student interaction determination method based on teaching events and student behavior provided in an embodiment of the present invention.

[0059] Figure 4 This is a schematic diagram illustrating the acquisition of student teaching status values ​​in a teacher-student interaction determination method based on teaching events and student behavior provided in an embodiment of the present invention.

[0060] Figure 5 This is a block structure diagram of an electronic device provided in an embodiment of the present invention.

[0061] Icons: 500 - Bus, 501 - Receiver, 502 - Processor, 503 - Transmitter, 504 - Memory, 505 - Bus Interface. Detailed Implementation

[0062] The present invention will now be described in detail with reference to the accompanying drawings.

[0063] Example 1

[0064] like Figure 1 As shown, this embodiment of the invention provides a method for determining teacher-student interaction based on teaching events and student behavior. The method includes:

[0065] S101: Obtain teaching events. Teaching events include the start time of the teaching event, the end time of the teaching event, and the teaching statement.

[0066] Among them, teaching statements are the statements made by the teacher during the time period from the start time of the teaching event to the end time of the teaching event.

[0067] S102: Based on teaching events, determine the relationship between teaching events and response time points to obtain multiple student response time points.

[0068] S103: Obtain a multimodal student state set at multiple student response time points. The multimodal student state set includes student face images, student body images, and student statements.

[0069] Among them, the multimodal student state set represents the state data corresponding to the student's learning behavior triggered by the teaching event.

[0070] S104: Based on a multimodal student state set at multiple student response time points, the states of teachers and students are determined within a set teaching time duration to obtain student teaching state values. Student teaching state values ​​represent the degree to which students have learned the teacher's teaching content.

[0071] Optionally, based on a multimodal student state set at multiple student response time points, the states of teachers and students are determined within a set teaching time length to obtain student teaching state values, including:

[0072] Based on student body images at multiple student response time points, the first feature vector and posture time points are obtained. The posture time points represent the time points in which students respond to teaching events.

[0073] The set of multimodal student states corresponding to the attitude time points is taken as the first set of multimodal student states.

[0074] Based on the first multimodal student state set, a second feature vector is obtained. The second feature vector represents the multimodal features of facial expressions, postures, and student statements at the time points of student feedback to teaching events.

[0075] Based on the second feature vector and the first feature vector, multimodal feature fusion is performed to obtain the student's teaching status value.

[0076] Using the above method, both individual and holistic approaches are employed, employing the interaction between the entire time period and individual posture time points to extract student feedback information on teaching events. Optionally, the entire time period (the set teaching time length) is the time interval from the start time to the end time of the teaching event, which can be set to a time interval of 5 to 10 seconds, including multiple time points. Student statements represent the content of the student's statements, which can be text content.

[0077] Optionally, based on student body images at multiple student response time points, a first feature vector and pose time points are obtained, including:

[0078] Based on the student's body image, joints are detected, resulting in a set of joint location locations. This set includes multiple joint category names and their corresponding locations.

[0079] In this embodiment, the OpenPose model is used to detect joints.

[0080] Multiple sets of joint point locations are obtained by corresponding to multiple student body images.

[0081] Based on the set of joint locations, a pose feature vector is obtained. Multiple pose feature vectors are obtained corresponding to multiple student body images.

[0082] Multiple pose feature vectors are input into the LSTM structure to obtain the first feature vector.

[0083] Using the above method, an LSTM structure is employed to extract the changes in the student's posture at multiple student response time points.

[0084] The attitude time points are obtained based on the set of multiple joint positions.

[0085] Optionally, attitude time points can be obtained based on multiple sets of joint positions, including:

[0086] Obtain the largest circle containing the set of joint positions to obtain the attitude circle.

[0087] In this process, multiple joint points in the set of joint points are connected in pairs, the polygon region with the largest area is taken as the polygon to be processed, and the smallest inscribed circle of the polygon to be processed is taken as the attitude circle.

[0088] Using the above method, the posture circle represents the area in which students extend their postures in response to the content taught by the teacher.

[0089] Calculate the area of ​​the attitude circle.

[0090] The posture area represents the area of ​​the region where the student is in an extended posture.

[0091] Multiple sets of joint positions correspond to multiple pose areas.

[0092] The student response time point corresponding to the largest posture area among multiple posture areas is taken as the posture time point.

[0093] Among them, the largest posture area represents the largest area of ​​the student's extended posture in response to the teacher's teaching content, and the posture time point is the student's response time point at this time.

[0094] By using the above method, the posture time points can be obtained to detect the posture that best matches the student's action over a period of time, and thus the student's feedback state to the teaching event can be found by using multimodal information at instantaneous time points.

[0095] Using the above method, since a student's extended posture over a wide range can indicate a significant impact of posture on learning attitude at specific time points, it is unnecessary to discriminate between different morphological changes in posture. Multimodal discrimination can then be performed at each posture time point.

[0096] Optionally, based on the set of keypoint locations, an attitude feature vector is obtained, including:

[0097] The association relationships between multiple joint categories are marked to obtain the joint connectivity matrix.

[0098] The relationships between multiple key point categories are stored in the database.

[0099] Multiple keypoint locations are input into the first feature extraction network to obtain keypoint location feature vectors. Multiple keypoint location feature vectors are obtained corresponding to multiple student response time points.

[0100] In this embodiment, the first feature extraction network is a fully connected neural network (FCNN).

[0101] In this process, multiple key points are input into the first feature extraction network in a fixed order.

[0102] The joint position feature vectors and the joint connectivity matrix are input into a graph convolutional network (GCN) to obtain pose feature vectors. Multiple joint position feature vectors correspond to multiple pose feature vectors.

[0103] Optionally, based on the first multimodal student state set, a second feature vector is obtained, including:

[0104] The student face images from the first multimodal student state set are input into the face feature extraction network to extract features and obtain the first face feature vector.

[0105] In this embodiment, the facial feature extraction network is a FaceNet model trained with multiple expression labels.

[0106] Based on the student body images in the first multimodal student state set, a first body feature vector is obtained through pose detection. The first body feature vector represents the pose of the limbs.

[0107] In this embodiment, the positions of multiple joints corresponding to the student's body image in the first multimodal student state set are input into a fully connected neural network (FCNN) to obtain a first body feature vector. The fully connected neural network (FCNN) is trained using posture labels such as raising hands, raising heads, and lowering heads.

[0108] The first facial feature vector and the first body feature vector are fused to obtain the second feature vector. In this embodiment, the first facial feature vector and the first body feature vector are concatenated to obtain the second feature vector. The number of elements in the second feature vector is equal to the sum of the number of elements in the first facial feature vector and the number of elements in the first body feature vector.

[0109] Optionally, the teaching statements and student statements are input into a Large Language Model (LLM) for keyword matching to obtain semantic matching values. In this embodiment, the semantic matching values ​​range from (0,1).

[0110] Using the above method, a Large Language Model (LLM) is employed to match the association between keywords in student statements and keywords in teaching statements, thereby determining whether students are understanding the content taught by the teacher.

[0111] Optionally, the semantic association value can be multiplied by the expansion constant and then divided by the length of the student's statement to obtain the student-teacher statement association value. In this embodiment, the expansion constant is 20.

[0112] Using the above method, since students' voice interactions may contain statements that are unrelated to the statements in the teaching events, we first perform statement matching and then consider the statements from multiple time points as a whole.

[0113] Optionally, based on the second feature vector and the first feature vector, the student's teaching state value is obtained, including:

[0114] If the correlation value between the student and teacher statements is greater than the correlation threshold, the student statement is input into the statement feature extraction network to obtain the statement feature vector.

[0115] In this embodiment, the association threshold is 16.

[0116] The sentence feature extraction network is the BERT model.

[0117] The first feature vector, the second feature vector, and the sentence feature vector are concatenated to obtain the concatenated vector.

[0118] The concatenated vector is input into the discriminant network to obtain the student's teaching status value.

[0119] The discriminant network is a combination of a fully connected neural network (FCNN) and softmax.

[0120] Among them, labeled student teaching status values ​​are used to train the discriminant network, the sentence feature extraction network, the graph convolutional network, and the LSTM structure.

[0121] Optionally, based on the teaching event, the relationship between the teaching event and the response time point is determined to obtain multiple student response time points, including:

[0122] The teaching time period is obtained based on the end time and start time of the teaching event.

[0123] The formula for calculating the teaching time period is as follows:

[0124]

[0125] in, The starting point of the teaching event. The point in time when the teaching event ends. This refers to the timeframe in advance during which students can respond before the end of the instructional statement. This refers to the buffer time that students have to think and react after a teaching event has ended. Indicates a teaching period.

[0126] In this embodiment, the advance time before the end of the teaching statement is 1 second, and the buffer time for the student to think and react after the teaching event ends is 10 seconds.

[0127] Multiple time points are evenly extracted from the teaching period to obtain multiple student response time points.

[0128] As another alternative embodiment:

[0129] S201: Establish a sequence of teaching events for teachers.

[0130] Acquire teacher voice recordings, courseware page numbers, screen operations, electronic lesson plans, or structured course knowledge graphs. Recognize and time-segment teacher voice recordings, and semantically match the voice text with lesson plan paragraphs, courseware pages, or knowledge point tags. Identify teaching events such as explanations, questions, prompts, feedback, and activity transitions to obtain:

[0131]

[0132] in, Used to identify knowledge points. This is a teacher-related event type. and These represent the start and end times of the event, respectively. To integrate semantic representations of text, courseware context, and event types, lesson plans and audio can be aligned using dynamic time warping, semantic similarity matching, or monotonic alignment models. For example:

[0133]

[0134] in, To satisfy the set of monotonically aligned paths that follow the order of classroom content progression, For the semantic representation of lesson plans or courseware, This is a timestamped text representation of the teacher's speech. The event recognition results can include confidence scores for knowledge points, event types, and time boundaries.

[0135] S202: Extract student multimodal features.

[0136] The various observations of student i are timestamped and feature-encoded to form:

[0137]

[0138] Each modal feature serves as observational evidence of the latent state, without directly mapping any single facial expression, posture, or physiological value to conclusions such as "acceptance / disapproval" or "seriousness / lack of seriousness."

[0139] S203: Estimate modal quality and causes of missing features.

[0140] Calculate the quality score for each student, each time step, and each modality:

[0141]

[0142] function The input can be a rule-based function, a learnable network, or a combination of both. Inputs may include signal-to-noise ratio, occlusion ratio, target tracking stability, device online status, timestamp drift, and feature anomaly degree. At least three types of markers are further defined: unusable markers due to device / occlusion, missing behavior markers indicating that the student did not produce the corresponding action, and low-quality markers indicating that the signal exists but is of low quality. During fusion, unusable modes are masked. The absence of behavior itself serves as semantic evidence. Low-quality signals are downweighted according to their quality scores.

[0143] S204: Establish individual student baselines.

[0144] Based on the baseline stage of students' history or current classroom lessons, calculate the individual mean and dispersion of each modality feature, and then standardize them:

[0145]

[0146] When the historical student sample is insufficient, a hierarchical contraction estimation between individual statistics and population statistics is used:

[0147]

[0148] in, It can increase with the increase of the effective sample size for each individual. In addition to the individual baseline, it can also calculate the contemporaneous distribution of the class under the same teaching event, which can be used to identify relative anomalies and avoid misjudging the common response of the whole class as individual anomalies.

[0149] S205: Construct the response window after the teacher event.

[0150] In response to the teacher incident Construct a time interval that includes the pre-event baseline and the post-event response:

[0151]

[0152] Different response delay kernels can be set or learned for different teacher event types and modalities. For example, a Gaussian kernel can be used:

[0153]

[0154] Where τ is the time difference between the student's observation of the start time of the relative event. This represents the typical response time delay of mode m to event type a. This indicates the degree of dispersion of the time delay distribution. The time delay kernel parameter can be learned from labeled data, or it can be initialized and then calibrated based on the class type. For continuous or overlapping teacher events, response assignment can be made through event semantic similarity, window overlap rate, and sparse matching constraints.

[0155] S206: Quality perception fusion under event conditions.

[0156] Using the semantic representation of teacher events as the query, and the multimodal features of students after baseline calibration as the key and value, while introducing time delay kernel and modal quality, modality-time weights are calculated:

[0157]

[0158] For observations where the device is unavailable, a masking term can be set before the softmax function. For valid observations where "behavior did not occur," their missing type embeddings can be preserved. This yields the event-conditional student response representation:

[0159]

[0160] This representation simultaneously encodes the semantics of teaching events, changes in students' relative individual baselines, the timing and location of responses, and the quality of observations, thereby reducing the impact of accidental co-occurrences and single-modal noise.

[0161] S207: Jointly estimate cognitive load, learning input, and response effectiveness.

[0162] Constructing two-dimensional potential states =( ),in For cognitive load, For learning input. Both are modeled separately, but correlation is allowed given event response, event semantics, and historical state:

[0163]

[0164] Calculate the state change using the posterior mean before and after the event:

[0165]

[0166] In one alternative implementation, the teacher event-student response coupling score is further calculated. and the probability of change of the target state The validity of the response is determined using the following formula:

[0167]

[0168] threshold , and It can be determined by the validation set, teacher annotation, or business tolerance requirements. Only when the state change meets the conditions of event semantics, response lag, and multimodal evidence is it judged as a valid response triggered by the teaching behavior. At the class level, students' state confidence can be weighted and aggregated to form the proportion and uncertainty of different load-input subgroups.

[0169] S208: Training, Calibration, and Inference.

[0170] When manual status annotation, task difficulty manipulation, answer accuracy, reaction time, or other weak supervision signals are available, joint optimization can be performed:

[0171]

[0172] in, Losses are monitored or weakly monitored for load and operational status. Match losses to teacher incidents with student responses. This is a state continuity constraint within non-event boundaries. Consistency constraints for different modalities of the same event. The loss is calibrated for confidence levels. During the training phase, modalities can be randomly masked or occlusions, noise, and temporal shifts can be injected to improve robustness to combinations of true missing values. During the inference phase, the complete posterior parameters are output instead of just hard labels.

[0173] Example 2

[0174] Based on the above-described method for determining teacher-student interaction based on teaching events and student behavior, this invention also provides a system for determining teacher-student interaction based on teaching events and student behavior, the system comprising:

[0175] The first acquisition module is used to acquire teaching events. Teaching events include the start time, end time, and teaching statements.

[0176] Based on teaching events, the relationship between teaching events and response time points is determined, and multiple student response time points are obtained.

[0177] The second acquisition module is used to acquire a multimodal student state set at multiple student response time points. The multimodal student state set includes student face images, student body images, and student statements.

[0178] The discrimination module is used to discriminate the states of teachers and students within a set teaching time period based on a multimodal set of student states at multiple student response time points, and obtain the student teaching state value. The student teaching state value represents the degree to which the student has learned the teacher's teaching content.

[0179] Regarding the systems and apparatus in the above embodiments, the specific methods by which each module performs its operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0180] This invention also provides an electronic device, such as... Figure 5As shown, it includes a memory 504, a processor 502, and a computer program stored in the memory 504 and executable on the processor 502. The processor 502 executes the steps of a teacher-student interaction determination method based on teaching events and student behavior.

[0181] Among them, Figure 5 In this document, a bus architecture (represented by bus 500) is used. Bus 500 may include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 502 and memory represented by memory 504. Bus 500 may also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 505 provides an interface between bus 500 and receiver 501 and transmitter 503. Receiver 501 and transmitter 503 may be the same element, i.e., a transceiver, providing a unit for communicating with various other devices over a transmission medium. Processor 502 is responsible for managing bus 500 and general processing, while memory 504 can be used to store data used by processor 502 during operation.

[0182] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the aforementioned teacher-student interaction determination method based on teaching events and student behavior.

[0183] This invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the teaching-oriented Socratic intelligent problem generation method described above.

[0184] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, this invention is not directed to any particular programming language. It should be understood that the contents of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0185] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0186] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the apparatus according to embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

Claims

1. A method for determining teacher-student interaction based on teaching events and student behavior, characterized in that, include: Acquire teaching events; The teaching event includes the start time of the teaching event, the end time of the teaching event, and the teaching statement; Based on the teaching events, the relationship between the teaching events and response time points is determined, and multiple student response time points are obtained. Obtain a multimodal student state set at multiple student response time points; the multimodal student state set includes student face images, student body images, and student statements; Based on the multimodal student state set at multiple student response time points, the states of teachers and students during a set teaching time are determined to obtain student teaching state values; the student teaching state values ​​represent the degree to which students have learned the teacher's teaching content.

2. The teacher-student interaction determination method based on teaching events and student behavior according to claim 1, characterized in that, The multimodal student state set based on the multiple student response time points is used to determine the states of teachers and students within a set teaching time length, obtaining student teaching state values, including: Based on the student body images at the multiple student response time points, a first feature vector and posture time points are obtained; the posture time points are the time points that characterize the student's posture in response to the teaching event; the first feature vector represents the change characteristics of the student's posture at the multiple student response time points. The set of multimodal student states corresponding to the posture time points is taken as the first set of multimodal student states. Based on the first multimodal student state set, a second feature vector is obtained; the second feature vector represents the multimodal features of facial expressions, postures, and student statements at the time point of student's response to teaching events. Based on the second feature vector and the first feature vector, multimodal feature fusion is performed to obtain the student's teaching status value.

3. The teacher-student interaction determination method based on teaching events and student behavior according to claim 2, characterized in that, The process of obtaining a first feature vector and pose time points from student body images based on the multiple student response time points includes: Based on the student's body image, joint points are detected to obtain a set of joint point locations; the set of joint point locations includes multiple joint point categories and corresponding joint point locations; multiple student body images correspond to multiple sets of joint point locations. Based on the set of joint positions, a posture feature vector is obtained; multiple posture feature vectors are obtained corresponding to multiple student body images; The multiple pose feature vectors are input into an LSTM structure to obtain the first feature vector; The attitude time points are obtained based on the set of multiple joint positions.

4. The teacher-student interaction determination method based on teaching events and student behavior according to claim 3, characterized in that, The process of obtaining attitude time points based on the set of multiple joint positions includes: Obtain the largest circle containing the set of joint point positions to obtain the attitude circle; Calculate the area of ​​the attitude circle to obtain the attitude area; Multiple sets of joint locations correspond to multiple pose areas; The student response time point corresponding to the largest of the multiple posture areas is taken as the posture time point.

5. The teacher-student interaction determination method based on teaching events and student behavior according to claim 3, characterized in that, The process of obtaining the attitude feature vector based on the set of joint positions includes: The association relationships between multiple joint categories are marked to obtain the joint connectivity matrix; Multiple key point locations are input into the first feature extraction network to obtain key point location feature vectors; multiple key point location feature vectors are obtained corresponding to multiple student response time points; The joint position feature vectors and the joint connectivity matrix are input into a graph convolutional network to obtain pose feature vectors; multiple pose feature vectors are obtained corresponding to multiple joint position feature vectors.

6. The method for determining teacher-student interaction based on teaching events and student behavior according to claim 2, characterized in that, The process of obtaining the second feature vector based on the first multimodal student state set includes: The student face images in the first multimodal student state set are input into a face feature extraction network to extract features and obtain a first face feature vector. Based on the student body images in the first multimodal student state set, a first body feature vector is obtained through pose detection; the first body feature vector represents the pose of the limbs. The first facial feature vector and the first body feature vector are fused to obtain the second feature vector.

7. The teacher-student interaction determination method based on teaching events and student behavior according to claim 2, characterized in that, Input the teaching statements and student statements into the large language model, perform keyword matching, and obtain semantic matching values; Multiply the semantic association value by the expansion constant and divide by the length of the student statement to obtain the student-teacher statement association value.

8. The method for determining teacher-student interaction based on teaching events and student behavior according to claim 7, characterized in that, The process of obtaining student teaching status values ​​based on the second feature vector and the first feature vector includes: If the correlation value between the student and teacher statements is greater than the correlation threshold, the student statement is input into the statement feature extraction network to obtain the statement feature vector; The first feature vector, the second feature vector, and the statement feature vector are concatenated to obtain the concatenated vector; The concatenated vector is input into the discriminant network to obtain the student's teaching status value.

9. The method for determining teacher-student interaction based on teaching events and student behavior according to claim 1, characterized in that, Based on the teaching event, the relationship between the teaching event and the response time point is determined, resulting in multiple student response time points, including: The teaching time period is obtained based on the end time and start time of the teaching event. The formula for calculating the teaching time period is as follows: Among them, the The starting point of the teaching event. The point in time when the teaching event ends. This refers to the timeframe in advance during which students can respond before the end of the instructional statement. This refers to the buffer time that students have to think and react after a teaching event has ended. Indicates a teaching period; Multiple time points are evenly extracted from the teaching period to obtain multiple student response time points.

10. A teacher-student interaction determination system based on teaching events and student behavior, characterized in that, include: The first acquisition module is used to acquire teaching events; The teaching event includes the start time of the teaching event, the end time of the teaching event, and the teaching statement; Based on the teaching events, the relationship between the teaching events and response time points is determined, and multiple student response time points are obtained. The second acquisition module is used to acquire a multimodal student state set at multiple student response time points; the multimodal student state set includes student face images, student body images, and student statements; The discrimination module is used to discriminate the status of teachers and students within a set teaching time length based on the multimodal student status set at the multiple student response time points, and obtain the student teaching status value; the student teaching status value represents the degree to which the student has learned the teacher's teaching content.

Images