Classroom teaching behavior analysis method and system based on multi-modal data fusion

Abstract

The application provides a classroom teaching behavior analysis method and system based on multi-modal data fusion, relates to the technical field of data analysis, and comprises the following steps: collecting a multi-modal classroom behavior data set, identifying interactive behavior events for discretization coding, establishing a behavior chain dependency graph model based on an interactive behavior event coding set, identifying a chain dependency path based on each target interactive behavior event, determining classroom behavior data to be fused, generating a fusion window weight according to the chain propagation time of the chain dependency path, weighting and fusing the classroom behavior data to be fused, obtaining classroom behavior fusion data, and obtaining an interactive behavior score index of each target interactive behavior event. The technical problem that the prior art lacks causal time sequence analysis of classroom multi-modal behavior fusion, resulting in low accuracy of quantitative evaluation of classroom teaching quality, is solved. The technical effect of accurate and reliable evaluation of classroom teaching quality is achieved.

Description

Technical Field

[0001] This invention relates to the field of data analysis technology, specifically to a method and system for analyzing classroom teaching behavior based on multimodal data fusion. Background Technology

[0002] With the development of artificial intelligence and educational technology, classroom teaching behavior analysis has become an important means to improve teaching quality. Currently, multimodal data is widely collected in classroom scenarios to understand teacher-student interaction patterns and learning processes. However, existing technologies still have significant limitations in multimodal behavior fusion. First, classroom behavior typically exhibits complex causal and temporal characteristics. For example, teacher questions may trigger student responses or discussions, but existing analysis methods often rely on static features or simple fusion strategies, lacking sophisticated modeling of causal relationships and temporal dependencies between behaviors. Second, existing quantitative methods struggle to accurately assess the actual contribution of various interactive behaviors to teaching effectiveness, resulting in insufficient accuracy and reliability of classroom quality assessment results, and failing to accurately reflect real classroom interactions.

[0003] Existing technologies suffer from a lack of causal and temporal analysis in the integration of multimodal classroom behaviors, making it difficult to accurately quantify the contribution of interactive behaviors and resulting in low accuracy in the quantitative evaluation of classroom teaching quality. Summary of the Invention

[0004] The purpose of this application is to provide a method and system for analyzing classroom teaching behavior based on multimodal data fusion, in order to solve the technical problem that existing technologies lack causal and temporal analysis in classroom multimodal behavior fusion, making it difficult to accurately quantify the contribution of interactive behaviors, resulting in low accuracy in the quantitative evaluation of classroom teaching quality.

[0005] In view of the above problems, this application provides a method and system for analyzing classroom teaching behavior based on multimodal data fusion.

[0006] The first aspect of this application provides a method for analyzing classroom teaching behavior based on multimodal data fusion. The method includes: collecting a multimodal classroom behavior dataset; identifying interactive behavior events in the multimodal classroom behavior dataset; discretizing and encoding the interactive behavior events to obtain an interactive behavior event encoding set; establishing a behavior chain dependency graph model based on the interactive behavior event encoding set; identifying chain dependency paths based on each target interactive behavior event according to the behavior chain dependency graph model; determining classroom behavior data of the modality to be fused according to the chain dependency paths; generating fusion window weights according to the chain propagation time of the chain dependency paths; weighted fusion of the classroom behavior data of the modality to be fused according to the fusion window weights to obtain fused classroom behavior data; and analyzing the fused classroom behavior data to obtain interactive behavior scoring indicators for each target interactive behavior event.

[0007] Optionally, the attribute information of the current classroom teaching behavior is identified, including classroom type, teaching mode type, and number of students; key temporal features of the classroom are extracted according to the attribute information; a first set of multimodal classroom behavior datasets under the key temporal features of the target classroom is obtained at a first data collection frequency; a second set of multimodal classroom behavior datasets not under the key temporal features of the classroom is obtained based on a second data collection frequency, wherein the first data collection frequency is greater than the second data collection frequency; based on the first set of multimodal classroom behavior datasets and the second set of multimodal classroom behavior datasets, a first set of interactive behavior event codes and a second set of interactive behavior event codes are obtained respectively.

[0008] Optionally, a local behavior chain dependency graph model and a global behavior chain dependency graph model are constructed according to the first set of interactive behavior event encoding sets and the second set of interactive behavior event encoding sets, respectively; based on the local behavior chain dependency graph model and the global behavior chain dependency graph model, the local chain dependency path and the global chain dependency path based on each target interactive behavior event are identified; according to the local chain dependency path and the global chain dependency path, local classroom behavior fusion data and global classroom behavior fusion data are obtained; according to the local classroom behavior fusion data and the global classroom behavior fusion data, the interactive behavior scoring index of each target interactive behavior event is recalculated.

[0009] Optionally, the local behavior chain dependency graph model is used to analyze the short-term window behavior triggering relationship of the first set of interactive behavior event encoding sets, and to analyze the edge weight composition of the first set of interactive behavior event encoding sets according to the short-term window behavior triggering relationship; the global behavior chain dependency graph model is used to analyze the long-term window behavior propagation relationship of the second set of interactive behavior event encoding sets, and to analyze the edge weight composition of the second set of interactive behavior event encoding sets according to the long-term window behavior propagation relationship.

[0010] Optionally, the interactive behavior events are obtained by analyzing the feature change rate of the multimodal classroom behavior dataset, comparing the feature change rate with a preset change rate threshold, and dividing the multimodal classroom behavior dataset into temporal behavior segments. The multimodal classroom behavior dataset includes at least visual behavior data, voice behavior data, posture features, and device interaction features. Multidimensional semantic codes are mapped to the interactive behavior events to construct an interactive behavior event code set. The multidimensional semantic codes include behavior event type, behavior role, behavior intensity, and behavior duration.

[0011] Optionally, the chain dependency path based on each target interaction behavior event is identified according to the behavior chain dependency graph model. Each graph node of the behavior chain dependency graph model corresponds to an interaction behavior event, and the edge weight is a weighted result based on the execution history transition probability of the latency decay factor, semantic association factor, and role dependency factor. The behavior chain dependency graph model is traversed to obtain the forward dependency graph nodes with a weight greater than a preset weight based on each target interaction behavior event. The forward dependency graph nodes are linked based on time constraints to obtain the chain dependency path based on each target interaction behavior event.

[0012] Optionally, the chain propagation time of each target interactive behavior event on the chain-dependent path with other nodes is calculated; a fusion window is generated based on the chain propagation time, and the corresponding classroom behavior data of the modality to be fused under the fusion window is obtained; a time decay function is constructed according to the chain propagation time, and a time weight is output based on the time decay function; the time weight is used to fuse the corresponding classroom behavior data of the modality to be fused under the fusion window.

[0013] Optionally, the modal correlation degree between the modal classroom behavior data group to be fused and each target interactive behavior event is calculated, and the modal correlation degree is obtained by calculating the mutual information value; modal weights are configured according to the modal correlation degree, and the joint weight of the modal weights and the time weights is calculated; the modal classroom behavior data to be fused under the fusion window is fused according to the joint weights.

[0014] Optionally, the increase rate of the number of interactive event triggers, the semantic consistency of interactive events, and the fluctuation of the quality of interactive events in the classroom behavior fusion data under the continuous fusion window are analyzed; the interactive behavior scoring index is obtained by calculating according to the increase rate of the number of interactive event triggers, the semantic consistency of interactive events, and the fluctuation of the quality of interactive events.

[0015] A second aspect of this application provides a classroom teaching behavior analysis system based on multimodal data fusion. The system includes: an event processing module for collecting a multimodal classroom behavior dataset, identifying interactive behavior events in the dataset, and discretizing and encoding the interactive behavior events to obtain an interactive behavior event encoding set; a path identification module for establishing a behavior chain dependency graph model based on the interactive behavior event encoding set, and identifying chain dependency paths based on each target interactive behavior event according to the model; a fusion data acquisition module for determining classroom behavior data of the modality to be fused based on the chain dependency paths, generating fusion window weights according to the chain propagation time of the chain dependency paths, and performing weighted fusion of the classroom behavior data of the modality to be fused according to the fusion window weights to obtain fused classroom behavior data; and a scoring index acquisition module for analyzing the fused classroom behavior data to obtain interactive behavior scoring indices for each target interactive behavior event.

[0016] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0017] The method provided in this application collects a multimodal classroom behavior dataset, identifies interactive behavior events in the dataset, discretizes and encodes these events to obtain an interactive behavior event encoding set, establishes a behavior chain dependency graph model based on the encoding set, identifies chain dependency paths based on each target interactive behavior event according to the model, determines classroom behavior data to be fused based on the chain dependency paths, generates fusion window weights according to the chain propagation time of the chain dependency paths, and performs weighted fusion of the classroom behavior data to be fused based on the fusion window weights to obtain fused classroom behavior data. The method analyzes the fused classroom behavior data to obtain interactive behavior scoring indicators for each target interactive behavior event. By constructing an interactive behavior chain dependency graph, selectively fusing relevant modal data using the chain propagation relationship between behaviors, and calculating the contribution score of interactive behaviors based on propagation intensity and influence range, the method achieves an effective and accurate quantitative assessment of classroom teaching quality, thereby improving the accuracy and reliability of classroom teaching behavior assessment.

[0018] The above description is merely an overview of the technical solution of this application. To better understand the technical means of this application and to facilitate its implementation according to the description, and to make the above and other objects, features, and advantages of this application more apparent, specific embodiments of this application are described below. It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent through the following description. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0020] Figure 1 A flowchart illustrating the classroom teaching behavior analysis method based on multimodal data fusion provided in this application.

[0021] Figure 2 A schematic diagram of the structure of the classroom teaching behavior analysis system based on multimodal data fusion provided in this application.

[0022] Figure labeling: Event processing module 11, Path recognition module 12, Fusion data acquisition module 13, Scoring index acquisition module 14. Detailed Implementation

[0023] This application provides a method and system for analyzing classroom teaching behavior based on multimodal data fusion. It addresses the technical problem of existing technologies lacking causal and temporal analysis in multimodal classroom behavior fusion, making it difficult to accurately quantify the contribution of interactive behaviors, thus leading to low accuracy in the quantitative assessment of classroom teaching quality. By constructing an interactive behavior chain dependency graph and combining the propagation time of the behavior chain with the correlation of multimodal features, it achieves accurate identification and fusion of classroom interactive behavior events, effectively quantifying the contribution of each interactive behavior, thereby achieving an accurate and reliable assessment of classroom teaching quality.

[0024] The technical solutions of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. It should be understood that the present invention is not limited to the exemplary embodiments described herein. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention. It should also be noted that, for ease of description, only the parts related to the present invention are shown in the accompanying drawings, not all of them.

[0025] Example 1, as Figure 1 As shown, this application provides a classroom teaching behavior analysis method based on multimodal data fusion, which includes:

[0026] Collect a multimodal classroom behavior dataset, identify the interactive behavior events in the multimodal classroom behavior dataset, and discretize and encode the interactive behavior events to obtain an interactive behavior event code set.

[0027] Furthermore, the interactive behavior events are discretized and encoded to obtain an interactive behavior event encoding set. The method includes: the interactive behavior events are obtained by analyzing the feature change rate of the multimodal classroom behavior dataset, comparing the feature change rate with a preset change rate threshold, and dividing the multimodal classroom behavior dataset into temporal behavior segments. The multimodal classroom behavior dataset includes at least visual behavior data, voice behavior data, posture features, and device interaction features. Multidimensional semantic codes are mapped to the interactive behavior events to construct an interactive behavior event encoding set. The multidimensional semantic codes include behavior event type, behavior role, behavior intensity, and behavior duration.

[0028] Specifically, by deploying multi-source data acquisition devices in the classroom environment to synchronously collect data on the classroom teaching process, a multimodal classroom behavior dataset is formed. The multimodal classroom behavior dataset includes at least visual behavior data, voice behavior data, posture features, and device interaction features. Specifically, high-definition cameras are used to capture visual behavior data of teachers and students from different angles, such as body movements and facial expressions. High-sensitivity microphones are used to collect classroom voice data, including the sounds of teachers lecturing, students asking and answering questions, and discussions. Wearable devices or image recognition technology are used to obtain key point posture features of teachers and students to represent teaching-related actions such as standing, turning, and pointing. At the same time, device interaction feature data, including blackboard operations, courseware switching, and answering questions, are obtained through teaching terminals, electronic whiteboards, or student interactive devices.

[0029] The data output by each acquisition device is time-stamped according to a unified system clock or global timestamp, and a preset unified time granularity is selected as the target time axis resolution. For modal data with a sampling frequency higher than the target time granularity, time downsampling is performed through resampling. For example, within the time window corresponding to the target time granularity, multiple frames of visual features or multiple segments of speech features are averaged, weighted, or representative values ​​are selected to form feature values ​​corresponding to the target time point. For modal data with a sampling frequency lower than the target time granularity, missing feature values ​​are estimated between adjacent time points through interpolation. For example, linear interpolation or a proximity-preserving strategy is used to complete the feature sequence. For device interaction feature data with irregular timestamps, discrete operation events are mapped to the nearest target time point or corresponding time window based on a time alignment algorithm. After time synchronization, to improve the stability and comparability of behavioral features, preprocessing operations are performed on the multimodal classroom behavior data. For continuous feature data such as vision, speech, and posture, smoothing filtering or moving average methods are used to suppress high-frequency interference introduced by changes in ambient lighting, background noise, or recognition jitter. To address the inconsistent acquisition scales of different individuals or devices, normalization processing, such as z-score or max-min normalization methods, is performed on each modality feature to map multimodal feature values ​​to a uniform numerical range, reducing individual and device differences. Simultaneously, statistical analysis is used to detect abnormal feature values. When feature values ​​deviate from historical distributions or local means exceed preset ranges, they are removed to suppress interference from occasional recognition errors or device malfunctions.

[0030] Under a unified time axis, the behavioral features extracted from each modality are subjected to sliding window analysis according to preset time segments. The rate of change of the corresponding behavioral features within adjacent time segments is calculated. The rate of change is used to characterize the degree of dynamic change of the behavioral features in a short period of time. For example, the sliding window analysis is performed with each preset time segment as 1 second. The behavioral features of a certain modality in time segment t i The value is F i In adjacent time segments t i+1 The value is F i+1 Then calculate the rate of change (F) i+1 -F iThe threshold is calculated as Δt / Δt, where Δt is the length of the time segment. When the rate of change of any modality is greater than or equal to a preset rate of change threshold, a significant classroom interaction is determined to have occurred at that time point. This time point is then used as the start or end boundary of the behavioral event, thereby segmenting continuous multimodal classroom behavior data on the time axis and identifying discrete interactive behavior events. The preset rate of change threshold can be obtained through statistical analysis of historical classroom data. For example, based on the distribution of the rate of change of behavioral features in a large number of labeled classroom samples, the threshold range for distinguishing between regular behavioral fluctuations and significant interactive behaviors can be determined using the mean plus standard deviation or percentile method. For example, the preset rate of change threshold can be set to the mean + 1.5 standard deviations or the 90th percentile position.

[0031] To ensure a unified computational representation for each interactive behavior event, a multidimensional semantic code is constructed for each event, and this code is discretized to create an interactive behavior event code set. The multidimensional semantic code includes at least four semantic dimensions: behavior event type, behavior role, behavior intensity, and behavior duration. The behavior event type characterizes the category attribute of the interactive behavior, such as asking a question, answering, explaining, or providing feedback. Its determination is based on a comprehensive assessment of key behavioral features extracted from multimodal classroom behavior data. This involves analyzing vocal activity states, intonation changes, keyword features, and sentence structure features in vocal behavior features, combined with body movement types, facial orientation, and the direction of interaction objects in visual behavior features, to semantically classify the interactive behavior. For example, when a teacher's vocal behavior is detected to have an upward intonation accompanied by body movements pointing towards students, the corresponding interactive behavior type can be determined as asking a question. When a student's continuous vocal output is detected accompanied by standing or raising a hand, it can be determined as answering a question. When vocal output is continuous and lacks obvious interactive feedback, while visual behavior is mainly focused on blackboard writing or courseware operation, it can be determined as explaining a question. When the voice output is short and accompanied by visual behaviors such as nodding or gesturing, it can be identified as a feedback behavior. The behavior role is used to distinguish the initiating entity, such as a teacher or student. In the visual modality, facial recognition, location relationships, or pre-labeled teacher and student areas are used to distinguish between teacher and student entities. In the voice modality, speaker separation and identity matching technologies are used to associate voice behaviors with corresponding entities. In the device interaction modality, the initiating entity is directly determined based on the account information or device ownership of the operating terminal. Combining the results of the multimodal entity recognition, when multiple modalities consistently identify the same entity within the same time period, the behavior role of the corresponding interactive behavior is determined to be either a teacher or a student.

[0032] The behavioral intensity is used to reflect the salience of interactive behavior in the corresponding modality, such as the amplitude of voice energy or the amplitude of body movements. In the voice modality, the energy characteristics, volume variation amplitude, and speech rate variation of the voice signal are first statistically analyzed within the duration of the behavioral event. For example, the energy of the voice frames is averaged over time, and the amplitude of energy variation between adjacent voice frames is calculated to obtain the voice behavior intensity index. In the visual and posture modality, the average displacement distance, maximum displacement amplitude, or number of movement changes per unit time of the key points are calculated by analyzing the key point posture sequence to characterize the amplitude and activity level of body movements. In the device interaction modality, the number of effective operations, operation trigger frequency, or operation response amplitude within the time window of the behavioral event are statistically analyzed to reflect the salience of interactive behavior at the device level. The duration of the behavior reflects the length of the interaction over time. It is determined based on the start and end boundaries of the interaction event on the timeline. When the rate of change of a feature exceeds a preset threshold and the interaction event is determined to have started, the corresponding time point is recorded as the start time of the event. When the rate of change of the behavior feature falls below the threshold or the behavior state undergoes a significant shift in subsequent time segments, the corresponding time point is recorded as the end time of the event. The duration of the interaction event is obtained by calculating the time difference between the start and end times.

[0033] For example, in a 45-minute math class, a multimodal classroom behavior dataset showed that 15 minutes into the lesson, the teacher suddenly increased their speaking speed and wrote rapidly on the blackboard. Visually, the teacher leaned forward, appearing more engaged. Analysis revealed that the feature change rate exceeded a preset threshold, identifying a new interactive behavior event. This event was then mapped with a multidimensional semantic code, determining the event type as "focused explanation," the role as "teacher," the intensity as "high," and the duration as "5 minutes," thus obtaining the interactive behavior event code.

[0034] By collecting multimodal classroom behavior datasets, including visual behavior data, voice behavior data, posture features, and device interaction features, a comprehensive analysis of classroom teaching behavior can be conducted from multiple dimensions. This avoids the limitations of single-modal data and more accurately reflects the actual situation of classroom teaching. Accurate identification and discretization of interactive behavior events provide an accurate and standardized data foundation for classroom teaching behavior analysis, ensuring that the entire analysis process is based on real and effective interactive behavior information, thereby improving the accuracy and reliability of the overall classroom teaching behavior analysis.

[0035] Furthermore, the method for collecting multimodal classroom behavior datasets also includes: identifying attribute information of current classroom teaching behavior, including classroom type, teaching mode type, and number of students; extracting key temporal features of the classroom according to the attribute information; acquiring a first set of multimodal classroom behavior datasets of the target classroom under the key temporal features at a first data acquisition frequency; acquiring a second set of multimodal classroom behavior datasets of the target classroom not under the key temporal features based on a second data acquisition frequency, wherein the first data acquisition frequency is greater than the second data acquisition frequency; and obtaining a first set of interactive behavior event codes and a second set of interactive behavior event codes based on the first set of multimodal classroom behavior datasets and the second set of multimodal classroom behavior datasets, respectively.

[0036] Specifically, collecting multimodal course behavior datasets can also identify the attribute information of current classroom teaching behavior. This attribute information includes classroom type, teaching mode type, and the number of students. Classroom type can be determined through the teaching schedule, course syllabus, or classroom recording content. For example, it can be labeled as lecture-based, experiment-based, or discussion-based, respectively. Teaching mode type can be determined through classroom environment and equipment information. For example, it can be identified as offline teaching, online teaching, or a hybrid mode by the presence of live video streaming, online interactive platforms, or blended learning terminals, respectively. The number of students can be obtained through visual modal face detection and counting algorithms. For example, face detection and tracking can be performed on classroom video frames, and the number of students in the teaching area can be counted to obtain the actual number of participants in the current classroom. For instance, in a 30-student offline lecture-based classroom, visual modal face detection identifies 28 students, teaching equipment records confirm the teaching mode as offline lecture-based, and the classroom type is determined to be lecture-based based on the course schedule information.

[0037] After acquiring classroom attribute information, key temporal features are extracted based on classroom type, teaching mode type, and number of students. These are time periods or teaching segments that significantly impact teaching effectiveness and student participation, such as teacher questioning, student discussion, or experimental operations. Specifically, key segments can be predefined based on classroom type. For example, in lecture-based classrooms, teacher questioning and blackboard explanation are key segments; in discussion-based classrooms, group discussion and student responses are key segments; and in experimental classrooms, experimental operations and demonstrations are key segments. The range of key temporal features is then dynamically adjusted based on the teaching mode type and number of students. For instance, in large lecture-based classrooms, teacher questioning may be short and scattered, while in small discussion-based classrooms, student discussions may be concentrated and longer. Specific time periods are extracted through real-time analysis of multimodal data. For example, in the visual modality, teacher gestures and changes in student hand-raising frequency are detected; in the speech modality, peak speech activity during teacher questioning or student responses is detected; and combined with operational events in the device interaction modality, specific time windows are determined as key temporal features of the classroom. For example, in lecture-based classrooms, by analyzing the teacher's vocal energy and the number of times students raise their hands, it is found that the teacher asks questions continuously and students interact actively from the 12th to the 15th minute. This time period is then identified as a key temporal feature of the classroom. In experimental classrooms, the period from the 30th to the 40th minute, when students conduct group experiments and frequently use the blackboard and operating equipment, is identified as a key temporal feature of the classroom.

[0038] For classroom segments exhibiting key temporal characteristics, multimodal classroom behavior data is collected at a high first data collection frequency. This includes sampling visual behavior data, vocal behavior data, postural features, and device interaction features every second. Visual behavior data includes teacher gesture amplitude and student hand-raising actions; vocal behavior data includes speech energy and speech rate changes; postural features include key point displacement and action frequency; and device interaction features include interactive whiteboard operation and the number of clicks during classroom quizzes. This results in the first set of multimodal classroom behavior datasets. For classroom segments with non-key temporal characteristics, corresponding multimodal data is collected at a lower second data collection frequency, such as sampling every 30 seconds, forming the second set of multimodal classroom behavior datasets. The first data collection frequency is higher than the second data collection frequency to ensure fine-grained capture of key behavioral information while reducing data redundancy and computational burden in non-key stages. Furthermore, multidimensional semantic encoding is performed on both the first and second sets of multimodal classroom behavior datasets. This multidimensional semantic encoding also includes behavioral event type, behavioral role, behavioral intensity, and behavioral duration, resulting in the first and second sets of interactive behavioral event encoding sets.

[0039] By distinguishing between key and non-key temporal characteristics in the classroom and employing different data collection frequencies, resources can be rationally allocated while ensuring comprehensive collection of classroom behavior data, thereby improving the efficiency and relevance of data collection. Simultaneously, two sets of interactive behavior event codes are obtained, enabling more accurate and detailed identification of interactive behavior events. This allows for clear differentiation of interactive behaviors of different types, roles, intensities, and durations, thus improving the effectiveness and accuracy of classroom teaching behavior analysis.

[0040] Furthermore, after obtaining the first set of interactive behavior event codes and the second set of interactive behavior event codes, the method further includes: constructing a local behavior chain dependency graph model and a global behavior chain dependency graph model according to the first set of interactive behavior event codes and the second set of interactive behavior event codes, respectively; identifying the local chain dependency path and the global chain dependency path based on the local behavior chain dependency graph model and the global behavior chain dependency graph model; obtaining local classroom behavior fusion data and global classroom behavior fusion data according to the local chain dependency path and the global chain dependency path; and recalculating the interactive behavior scoring index of each target interactive behavior event according to the local classroom behavior fusion data and the global classroom behavior fusion data.

[0041] Furthermore, a local behavior chain dependency graph model and a global behavior chain dependency graph model are constructed respectively. The method includes: wherein the local behavior chain dependency graph model is used to analyze the short-term window behavior triggering relationship of the first set of interactive behavior event encoding sets, and analyzes the edge weight composition of the first set of interactive behavior event encoding sets according to the short-term window behavior triggering relationship; the global behavior chain dependency graph model is used to analyze the long-term window behavior propagation relationship of the second set of interactive behavior event encoding sets, and analyzes the edge weight composition of the second set of interactive behavior event encoding sets according to the long-term window behavior propagation relationship.

[0042] Specifically, a local behavior chain dependency graph model is constructed for the first set of interactive behavior event codes. This model is used to analyze behavior triggering relationships within a short time window. When constructing the local behavior chain dependency graph model, each event in the first set of interactive behavior event codes is used as a node in the graph, such as a teacher asking a question, a student answering, or a blackboard writing operation. By analyzing the sequence of events and causal relationships within the short time window, the edges between nodes are determined; these edges represent behavior triggering relationships. For example, within a short time window, after a teacher asks a question (event A) and a student quickly answers (event B), there exists an edge in the local behavior chain dependency graph model pointing from event A to event B.

[0043] In local behavior chain dependency graph models, edge weights are used to quantify the triggering strength between events within a short time window. Their calculation can be based on a combination of latency decay factors, semantic association factors, role dependency factors, and historical triggering probabilities. For example, by calculating the time interval Δt between event A and event B, a time decay function, such as the exponential decay function w, can be used. t =e -αΔt Where α is the attenuation coefficient, set to 1~5s - Within the range of ¹, the time interval is converted into a time delay attenuation factor. The shorter the time interval, the higher the time delay attenuation factor value. Simultaneously, the semantic correlation between event A and event B is analyzed, such as the matching degree of question-answer or operation-feedback. The semantic correlation factor w is obtained through keyword matching and semantic vector similarity calculation. s For example, by preprocessing the text content of events A and B, including removing stop words and performing stemming, and using word frequency statistics, words that appear frequently and are representative in the event text are selected as keywords to obtain keyword sets for events A and B. For example, in the teacher's question "Please explain the process of photosynthesis," the keywords could be photosynthesis, explanation, and process. The overlap of keywords between the two events is calculated by dividing the number of elements in the intersection of the two keyword sets by the number of elements in the union of the two keyword sets, and then normalized to obtain the semantic association factor for keyword matching. The historical transition probability w is obtained based on the frequency of event A triggering event B in previous classroom samples. h In a sample of history lessons, count the number of times N occurs after event A within a short time window Δt. AB And the total number of times event A occurs, N A Calculate the historical trigger probability w based on frequency. h =N AB / N A This reflects the probability that event A will trigger event B. The three types of weights are combined using a weighted or multiplicative method to obtain the final edge weight, W1 = w t ×w s ×w h .

[0044] For the second set of interactive behavior event codes, a global behavior chain dependency graph model is constructed. This model is used to analyze the behavior propagation relationship within a long-term window. The long-term window encompasses the entire lesson's time frame, showcasing the propagation and evolution of classroom behavior over a longer time dimension. Each event in the second set of interactive behavior event codes is used as a graph node, such as a teacher asking a question, a student answering, a group discussion, or a blackboard writing operation. Then, edges in the graph are determined based on the propagation logic between events within the long-term window. Specifically, the event sequence is traversed according to the lesson's time order. For events A and B, if event B occurs within a certain time frame after event A and the two have thematic, semantic, or role-related connections, an edge is established between event A and event B, indicating that event A may propagate or trigger event B. Next, the propagation pattern of events is analyzed. For example, a teacher asking a question may guide multiple students to answer, or a group discussion may spread to other groups. The validity of the edge is determined by statistically analyzing the frequency of event A triggering event B and the role interaction patterns in historical classroom data. Finally, the edge weights are determined by weighting the propagation probability, semantic relevance, and role interaction patterns between events over a long time span, in order to characterize the macro-interaction structure and the overall classroom atmosphere. Specifically, this is achieved by statistically analyzing the propagation frequency or probability p of event A to event B throughout the entire lesson. AB For example, teachers can ask questions to guide different students to answer or engage in group discussions, forming long-term propagation paths. This involves calculating the semantic consistency between events. AB Similarly, it can be calculated based on event topic matching degree and keyword similarity, and then the role interaction mode weight r is introduced. AB This is used to reflect the typicality or importance of teacher-student, student-student, or group interactions. It can be obtained through historical classroom data or behavioral pattern statistics. The three types of weights are weighted and fused to obtain the edge weights W2=β1×p of the global behavioral chain dependency graph model. AB +β2×s AB +β2×r AB β1, β2, and β3 are adjustment parameters used to control the proportion of influence of each weight in the final edge weight. They can be set according to the classroom analysis objectives, such as 0.4, 0.3, and 0.3 respectively, to ensure that the propagation probability is slightly dominant, while taking into account semantic and role factors.

[0045] In the local behavior chain dependency graph model, for each target interactive behavior event, such as a single teacher question or student answer, nodes and their forward and backward edges are traversed using depth-first search or breadth-first search. Triggering events are traced sequentially along the edges of the short-term window to identify complete local chain dependency paths. These local chain dependency paths represent the interaction effect of the target interactive behavior event at a micro-timescale, reflecting a series of triggering and response processes surrounding a specific interactive behavior event within the short-term window. The local chain dependency paths characterize the micro-interaction effects of a single question or answer. Similarly, in the global behavior chain dependency graph model, for each target interactive behavior event, the propagation sequence and path between events are recorded along the propagation edges of the long-term window to obtain the global chain dependency path, which reflects the macro-interaction trend and overall classroom atmosphere of the entire lesson. For example, in a local behavior chain dependency graph model, a teacher asks a question (event A), which triggers a student's answer (event B), and then triggers a blackboard writing operation (event C). The local chain dependency path is A→B→C. In a global behavior chain dependency graph model, the teacher asks questions multiple times, which trigger different students' answers and group discussions, forming the path A1→B1→C1→B2→C2, which is the global chain dependency path.

[0046] The multimodal behavioral features corresponding to each interactive event in the local chain dependency path and the global chain dependency path are used as the fusion object, including voice intensity, visual motion amplitude, posture change amount, and device interaction frequency. Each target interactive event is regarded as a node feature vector on the path. Based on the edge weights between adjacent nodes in the local chain dependency path and the global chain dependency path, as well as the temporal relationship between the event and the target interactive event, the node features are weighted and aggregated. Among them, the local classroom behavior fusion data focuses on the short-term triggering effect, and a time decay factor can be introduced to assign smaller weights to nodes in the path that are farther away from the target event. The global classroom behavior fusion data focuses on the long-term propagation effect, and the nodes in the path can be weighted and accumulated based on the propagation probability and role interaction mode. Specifically, let the target interactive event be e0, its corresponding local chain dependency path be {e0,e1…en}, the feature vector of each event be Fi, and the edge weight be W1. Then the local classroom behavior fusion data can be represented as: For the global chain dependency path {e0, e1…em}, with edge weight W2, the global classroom behavior fusion data is represented as follows: Where n is the number of local short-term window dependent events and m is the number of global long-term window dependent events.

[0047] Then, based on the local and global classroom behavior fusion data, the interaction behavior score index for each target interactive behavior event is recalculated to comprehensively reflect the role of the event at both the micro and macro levels. By weighting and fusing the local and global fusion data indicators according to a set weight, the interaction behavior score index for each target interactive behavior event is obtained. The interaction behavior score index S = γ × F1(e0) + (1-γ) × F2(e0), where γ is the local and global fusion weight adjustment parameter, which can be set according to the analysis objective. For example, setting γ = 0.5 indicates that the weights of the local and global classroom behavior fusion data are equal. The final score index can be used to evaluate the comprehensive influence and contribution of teacher questioning, student answering, or other interactive behaviors in the classroom, providing a quantitative basis for classroom behavior analysis and teaching quality evaluation.

[0048] By constructing local and global behavioral chain dependency graph models, we can deeply analyze the relationships between classroom interactive behavioral events from both micro and macro perspectives, obtaining more comprehensive and detailed information about classroom behavior. Recalculating the interactive behavior scoring indicators based on this information allows them to more accurately and objectively reflect the true state of classroom interaction, further improving the comprehensiveness, accuracy, and reliability of classroom teaching behavior analysis.

[0049] A behavior chain dependency graph model is established based on the set of interactive behavior event codes, and the chain dependency path based on each target interactive behavior event is identified according to the behavior chain dependency graph model.

[0050] Furthermore, the chain dependency graph model is used to identify chain dependency paths based on each target interaction event. Each graph node in the chain dependency graph model corresponds to an interaction event, and the edge weight is a weighted result of the execution history transition probability based on the latency decay factor, semantic association factor, and role dependency factor. The chain dependency graph model is traversed to obtain forward dependency graph nodes with weights greater than a preset weight for each target interaction event. The forward dependency graph nodes are linked based on time constraints to obtain the chain dependency path based on each target interaction event.

[0051] Specifically, a behavior chain dependency graph model is established based on the set of interactive behavior event codes. Each discretized interactive behavior event is treated as a node in the behavior chain dependency graph model, where each node corresponds to a multidimensional semantically encoded event. The multidimensional semantic encoding includes at least the behavior event type, behavior role, behavior intensity, and behavior duration, to fully reflect the attributes of interactive behaviors in the classroom. Directed edges are constructed between nodes based on the temporal and semantic relationships of the interactive behavior events to represent the possible triggering or propagation dependencies between behavior events. To quantify the strength of dependencies between different events, an edge weight is assigned to each edge, and the edge weight is based on a delay decay factor w. t1 Semantic association factor w s1 Role dependency factor wr and historical transition probability w h1 We obtain W3 = w by weighted fusion. t1 ×w s1 ×w r1 ×w h1 Among them, the time delay decay factor is used to characterize the closeness of two events in the time dimension. For example, the shorter the event interval, the smaller the decay. The semantic association factor is used to measure the degree of matching between event type and behavior content. The role dependency factor is used to reflect the typicality of different role combinations such as teacher-student and student-student in classroom interaction. The role dependency factor wr is obtained by statistical analysis based on historical classroom behavior data. The role dependency factor wr is used to quantify the typicality and dependency strength of different behavioral role combinations in classroom interaction. It can be achieved by classifying and statistically analyzing interactive behavior events according to behavioral roles in a large number of labeled classroom samples, such as teacher→student, student→teacher, student→student, group→whole class, etc. The frequency or conditional probability of event A triggering event B in various role combinations under the same or similar teaching scenarios is calculated. Then, the statistical results are normalized so that the role dependency factor value is between 0 and 1. The higher the frequency of occurrence and the more in line with the conventional classroom interaction pattern, the larger the role dependency factor value. For example, in lecture-based classrooms, the frequency of teacher-led questioning triggering student responses is significantly higher than in random student interactions. Therefore, a higher role dependency factor, such as 0.85, can be assigned to the teacher-student combination. This role dependency factor effectively reflects the influence of role structure on the dependency relationship of classroom interaction behaviors. The historical transition probability is obtained by statistically analyzing the frequency of event A triggering event B in a historical classroom sample.

[0052] After constructing the behavior chain dependency graph model, the chain dependency path based on each target interaction behavior event is identified according to the behavior chain dependency graph model. Specifically, starting from the node corresponding to the target interaction behavior event, a forward traversal is performed along the directed edges in the behavior chain dependency graph model whose weights are greater than a preset weight threshold. Nodes in the forward dependency graph that have a significant dependency relationship with the target interaction behavior event are filtered out. The preset weight threshold is used to filter out occasional or weakly related behaviors, retaining only interaction events that have a real impact on the target event. For example, it is set to 0.6 to ensure that the interaction behavior events in the path have high triggering credibility. Simultaneously, a time constraint is introduced during the traversal process, allowing only events that occur consecutively within a preset time window to be linked, avoiding invalid associations caused by excessively long time spans. Event nodes that meet the weight threshold and time constraint are connected in chronological order to form a chain dependency path based on each target interaction behavior event. For example, when the teacher asks a question as the target node, events such as student answers and blackboard supplements are linked sequentially along edges with a weight greater than 0.6, forming a chain dependency path of teacher question → student answer → teacher explanation reinforcement, thus reflecting the actual interactive influence range of the target event in the classroom.

[0053] By constructing a behavioral chain dependency graph model and identifying chain dependency paths based on target interactive behavioral events, isolated and discrete classroom interactive behaviors are organized into structured behavioral chains with causal and propagation relationships. This effectively reflects the triggering logic and evolution process of interactive behaviors in the target classroom. By introducing weight thresholds and time constraints, noise interference can be suppressed while ensuring the accuracy of the analysis, further improving the stability and accuracy of classroom behavior analysis results.

[0054] The classroom behavior data of the modalities to be merged is determined based on the chain dependency path. The fusion window weights are generated according to the chain propagation time of the chain dependency path. The classroom behavior data of the modalities to be merged is weighted and merged according to the fusion window weights to obtain the classroom behavior fusion data.

[0055] Furthermore, the method for generating fusion window weights according to the chain propagation time of the chain-dependent path includes: calculating the chain propagation time of each target interactive behavior event on the chain-dependent path with other nodes; generating a fusion window based on the chain propagation time, and obtaining the corresponding classroom behavior data of the modality to be fused under the fusion window; constructing a time decay function according to the chain propagation time, and outputting time weights based on the time decay function; and using the time weights to fuse the corresponding classroom behavior data of the modality to be fused under the fusion window.

[0056] Specifically, using the target interactive behavior event as a reference node, other interactive behavior events with causal dependencies on the target interactive behavior event are identified along its corresponding chain dependency path. The visual behavior data, voice behavior data, posture feature data, and device interaction feature data corresponding to these interactive behavior events in the multimodal classroom behavior dataset are then identified as the classroom behavior data to be fused. The chain propagation time between each interactive behavior event on the chain dependency path and the target interactive behavior event, i.e., the time difference of event occurrence, is calculated to characterize the propagation distance of the behavior's influence over time. Given an established causal relationship, a shorter chain propagation time indicates higher information freshness and relevance of the interactive behavior event to the current target state. Therefore, a fusion window is generated based on the chain propagation time. This involves extending forward along the time axis from the time of the target event occurrence to cover the time range of each event in the chain dependency path, thereby obtaining the corresponding multimodal classroom behavior data within the fusion window. The fusion window is a time interval that determines the data range considered during data fusion. For example, if the fusion window size is set to 10 seconds, and event C is taken as the target event, based on the chain propagation time calculated earlier, the relevant events within 10 seconds are traced back. Assuming that event B occurs 5 seconds before event C and event A occurs 5 seconds before event B, then the fusion window will cover the relevant data of the three events, event A, event B and event C, and obtain the corresponding classroom behavior data of the modality to be fused under the fusion window.

[0057] Based on this, a time decay function can be constructed according to the chain propagation time, for example, using the exponential decay function w=e -αΔti , where Δ ti Let α be the chain propagation time and α be the decay coefficient, adjusted according to the actual pace of the class and the characteristics of the data, such as setting α to 0.3. By constructing a time decay function, the chain propagation time is mapped to a time weight; the shorter the chain propagation time, the greater the corresponding time weight. The time weights are used to perform weighted fusion of multimodal classroom behavior data within the fusion window. Specifically, the multimodal behavior features corresponding to each time segment within the fusion window are represented as feature vectors, including speech energy, speech rate changes, visual motion amplitude, posture key point displacement, and device interaction frequency. Then, the feature vector of each time segment is weighted and calculated with its corresponding time weight, so that behavior data with shorter chain propagation times and larger time weights occupy a higher proportion in the fusion result, while the contribution of data with longer chain propagation times decreases accordingly. A weighted summation method is used, multiplying the feature vectors of each time segment by their time weights and then summing them to obtain fused classroom behavior data at a unified scale.

[0058] By generating fusion window weights based on chain propagation time according to chain dependency paths and performing multimodal weighted fusion, the causal order and temporal correlation between behaviors can be fully considered when fusing multi-source classroom behavior data. This avoids assigning excessive weights to behaviors with long time distances and low correlation, thereby effectively improving the interpretability and discrimination accuracy of classroom behavior fusion data for target interactive behaviors. It also enhances the temporal rationality and semantic consistency of multimodal data fusion results, and ultimately achieves a reliable and accurate quantitative assessment of classroom teaching quality.

[0059] Furthermore, the method of fusing the classroom behavior data of the modalities to be fused under the fusion window using the time weights further includes: calculating the modal correlation degree between the classroom behavior data group of the modalities to be fused and each target interactive behavior event, wherein the modal correlation degree is obtained by calculating mutual information value; configuring modal weights according to the modal correlation degree, and calculating the joint weight of the modal weights and the time weights; and fusing the classroom behavior data of the modalities to be fused under the fusion window according to the joint weights.

[0060] Specifically, the process of fusing classroom behavior data of the modalities to be fused under the fusion window using time weights can also be as follows: For the group of classroom behavior data of the modalities to be fused within the fusion window, calculate the mutual information value between each modal behavior feature and the target interactive behavior event. First, the target interactive behavior event is represented as an event label or event intensity sequence, such as whether it has occurred, event type encoding, or event score. Then, each modal behavior feature is represented as time series data within the fusion window according to a unified time granularity. For continuous modal features, such as speech energy, movement amplitude, and device operation frequency, discretization or probability distribution modeling is performed, for example, using equal-width binning, equal-frequency binning, or kernel density estimation methods to construct the joint probability distribution and marginal probability distribution of modal feature random variables and target event random variables. The mutual information value between each modal feature random variable X and the target interactive behavior event variable Y is calculated according to the mutual information formula, and the calculation result is used as a modal correlation index.

[0061] Modal weights are assigned to different modalities based on the relative magnitudes of their mutual information values. By normalizing the mutual information values ​​of different modalities, modal weights reflecting the explanatory power of each modality for the target interaction behavior can be obtained. The larger the mutual information value, the higher the correlation between the corresponding modality and the target interaction behavior in the analysis. After obtaining the modal weights w... m and the time weight w obtained based on chain propagation time ti Subsequently, when weighting and fusing the corresponding multimodal classroom behavior data within the fusion window based on the joint weight, the behavioral features of each time segment and each modality within the fusion window are first aligned and represented as feature vectors of a unified dimension. Then, the joint weight w is calculated for each modality and each time segment. j=w ti ×w m This approach assigns greater weight to data that is closer in time to the target interaction behavior and has a higher modal correlation. Based on this, a weighted summation method is used to fuse multimodal features. The feature vectors of each modality are multiplied by their corresponding joint weights and then summed. The joint weights are then normalized to obtain the final classroom behavior fusion data.

[0062] By introducing modal correlation on top of time weights and constructing joint weights, the problem of simply treating different modal data equally during the fusion process can be effectively avoided. This allows modalities that contribute more to the target interactive behavior to receive greater influence weights, thereby improving the discriminative ability and accuracy of classroom behavior fusion data. Furthermore, this joint weighting mechanism fully considers the freshness of behavioral data in the time dimension and its relevance to the target interactive behavior in terms of semantics and function, ensuring that the generated classroom behavior fusion data can accurately and comprehensively reflect classroom behavior characteristics, thereby improving the effectiveness and reliability of quantitative assessment of classroom teaching quality.

[0063] The interactive behavior scoring index for each target interactive behavior event is obtained by analyzing the classroom behavior fusion data.

[0064] Furthermore, the interactive behavior scoring index for each target interactive behavior event is obtained by analyzing the classroom behavior fusion data. The method includes: analyzing the increase rate of the number of interactive event triggers, the semantic consistency of interactive events, and the fluctuation of the quality of interactive events in the classroom behavior fusion data under a continuous fusion window; and calculating the interactive behavior scoring index according to the increase rate of the number of interactive event triggers, the semantic consistency of interactive events, and the fluctuation of the quality of interactive events.

[0065] Specifically, after obtaining the classroom behavior fusion data, for each target interactive behavior event, a time-series analysis is performed on the fusion data within a continuous fusion window to extract multi-dimensional evaluation indicators that can characterize the interaction effect. First, the increase rate of the number of interactive event triggers is analyzed to reflect the driving effect of the target interactive behavior on the subsequent classroom interaction activity. This is calculated by comparing the changes in the number of interactive behavior events triggered within adjacent fusion windows, such as statistically analyzing the increase in the number of student answers, discussions, or feedback events across multiple fusion windows after the target event occurs. Second, the semantic consistency of interactive events is analyzed to measure the coherence and teaching focus of continuous interactive behaviors at the content level. This can be achieved by calculating the semantic similarity of voice text, behavior tags, or event topic vectors in the fusion data, such as using cosine similarity calculation. Higher semantic consistency indicates that the interaction surrounding the target interactive behavior is more focused and more aligned with the teaching objectives. Furthermore, the fluctuation of interactive event quality is analyzed to reflect the stability and trend of interactive behavior quality over time. This is quantified by comparing the variance or fluctuation range of quality indicators such as behavior intensity, participation, or answer accuracy within a continuous fusion window. The smaller the variance, the more stable the interaction quality.

[0066] Finally, the increase rate of the number of interactive event triggers, the semantic consistency of interactive events, and the fluctuation of interactive event quality changes are comprehensively calculated according to preset weights to obtain the interactive behavior score index of the target interactive behavior event. The specific calculation method is as follows: Interactive behavior score index S = a × increase rate of the number of interactive event triggers + b × semantic consistency of interactive events + c × fluctuation of interactive event quality changes, where a, b, and c are weight parameters that can be set according to the teaching evaluation needs, such as 0.4, 0.3, and 0.3 respectively.

[0067] By jointly analyzing multi-dimensional indicators such as the increase rate of interactive event triggers, semantic consistency, and quality change fluctuations based on classroom behavior fusion data, we can quantitatively evaluate the actual teaching effect of each target interactive behavior event. This avoids evaluation bias caused by relying solely on a single behavioral feature or instantaneous performance. At the same time, it can reflect the effect of target interactive behavior events at the short-term interaction level and comprehensively reflect their influence in the continuous evolution of classroom behavior. This provides an objective, accurate, and reliable quantitative basis for classroom interaction quality analysis, teaching behavior optimization, and teaching effect feedback, thereby further improving teaching quality.

[0068] Example 2, based on the same inventive concept as the classroom teaching behavior analysis method based on multimodal data fusion in the previous examples, such as... Figure 2 As shown, this application provides a classroom teaching behavior analysis system based on multimodal data fusion, wherein the classroom teaching behavior analysis system based on multimodal data fusion includes:

[0069] Event processing module 11 is used to collect a multimodal classroom behavior dataset, identify interactive behavior events in the multimodal classroom behavior dataset, and discretize and encode the interactive behavior events to obtain an interactive behavior event encoding set; path identification module 12 is used to establish a behavior chain dependency graph model based on the interactive behavior event encoding set, and identify the chain dependency path based on each target interactive behavior event according to the behavior chain dependency graph model; fusion data acquisition module 13 is used to determine the classroom behavior data to be fused based on the chain dependency path, generate fusion window weights according to the chain propagation time of the chain dependency path, and perform weighted fusion of the classroom behavior data to be fused based on the fusion window weights to obtain classroom behavior fusion data; scoring index acquisition module 14 is used to analyze the classroom behavior fusion data to obtain the interactive behavior scoring index for each target interactive behavior event.

[0070] Furthermore, the event processing module 11 is also used to: identify the attribute information of the current classroom teaching behavior, including classroom type, teaching mode type, and number of teaching objects; extract key temporal features of the classroom according to the attribute information, obtain a first set of multimodal classroom behavior datasets of the target classroom under the key temporal features at a first data acquisition frequency, and obtain a second set of multimodal classroom behavior datasets of the target classroom not under the key temporal features based on a second data acquisition frequency, wherein the first data acquisition frequency is greater than the second data acquisition frequency; and obtain a first set of interactive behavior event codes and a second set of interactive behavior event codes based on the first set of multimodal classroom behavior datasets and the second set of multimodal classroom behavior datasets.

[0071] Furthermore, the system is also used to: construct a local behavior chain dependency graph model and a global behavior chain dependency graph model according to the first set of interactive behavior event encoding sets and the second set of interactive behavior event encoding sets, respectively; identify the local chain dependency path and the global chain dependency path based on the local behavior chain dependency graph model and the global behavior chain dependency graph model; obtain local classroom behavior fusion data and global classroom behavior fusion data according to the local chain dependency path and the global chain dependency path; and recalculate the interactive behavior scoring index of each target interactive behavior event according to the local classroom behavior fusion data and the global classroom behavior fusion data.

[0072] Furthermore, the system is also used for: the local behavior chain dependency graph model to analyze the short-term window behavior triggering relationship of the first set of interactive behavior event encoding sets, and to analyze the edge weight composition of the first set of interactive behavior event encoding sets according to the short-term window behavior triggering relationship; the global behavior chain dependency graph model to analyze the long-term window behavior propagation relationship of the second set of interactive behavior event encoding sets, and to analyze the edge weight composition of the second set of interactive behavior event encoding sets according to the long-term window behavior propagation relationship.

[0073] Furthermore, the event processing module 11 is also used for: obtaining the interactive behavior event by analyzing the feature change rate of the multimodal classroom behavior dataset, comparing the feature change rate with a preset change rate threshold to divide the multimodal classroom behavior dataset into temporal behavior segments, wherein the multimodal classroom behavior dataset includes at least visual behavior data, voice behavior data, posture features and device interaction features; mapping multidimensional semantic codes to the interactive behavior event, constructing an interactive behavior event code set, wherein the multidimensional semantic code includes behavior event type, behavior role, behavior intensity and behavior duration.

[0074] Furthermore, the path identification module 12 is also used to: identify chain dependency paths based on each target interaction behavior event according to the behavior chain dependency graph model, wherein each graph node of the behavior chain dependency graph model corresponds to an interaction behavior event, and the edge weight is a weighted result based on the probability of execution history transition based on the latency decay factor, semantic association factor, and role dependency factor; traverse the behavior chain dependency graph model to obtain forward dependency graph nodes with a weight greater than a preset weight based on each target interaction behavior event; and link the forward dependency graph nodes based on time constraints to obtain chain dependency paths based on each target interaction behavior event.

[0075] Furthermore, the fusion data acquisition module 13 is also used to: calculate the chain propagation time of each target interactive behavior event on the chain dependency path and other nodes; generate a fusion window based on the chain propagation time, and obtain the corresponding classroom behavior data of the modality to be fused under the fusion window; construct a time decay function according to the chain propagation time, and output a time weight based on the time decay function; and use the time weight to fuse the corresponding classroom behavior data of the modality to be fused under the fusion window.

[0076] Furthermore, the data fusion acquisition module 13 is also used to: calculate the modal correlation degree between the modal classroom behavior data group to be fused and each target interactive behavior event, wherein the modal correlation degree is obtained by calculating mutual information value; configure modal weights according to the modal correlation degree, and calculate the joint weight of the modal weights and the time weights; and fuse the modal classroom behavior data to be fused under the fusion window according to the joint weights.

[0077] Furthermore, the scoring index acquisition module 14 is also used to: analyze the increase rate of the number of interactive event triggers, the semantic consistency of interactive events, and the fluctuation of the quality of interactive events in the classroom behavior fusion data under the continuous fusion window; and calculate the interactive behavior scoring index according to the increase rate of the number of interactive event triggers, the semantic consistency of interactive events, and the fluctuation of the quality of interactive events.

[0078] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The classroom teaching behavior analysis method and specific examples based on multimodal data fusion in the foregoing embodiment one are also applicable to the classroom teaching behavior analysis system based on multimodal data fusion in this embodiment. Through the foregoing detailed description of the classroom teaching behavior analysis method based on multimodal data fusion, those skilled in the art can clearly understand the classroom teaching behavior analysis system based on multimodal data fusion in this embodiment. Therefore, for the sake of brevity, it will not be described in detail here.

[0079] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0080] Obviously, those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for analyzing classroom teaching behavior based on multimodal data fusion, characterized in that, The method includes: Collect a multimodal classroom behavior dataset, identify the interactive behavior events in the multimodal classroom behavior dataset, and discretize and encode the interactive behavior events to obtain an interactive behavior event code set; A behavior chain dependency graph model is established based on the set of interactive behavior event codes, and the chain dependency path based on each target interactive behavior event is identified according to the behavior chain dependency graph model. The classroom behavior data of the modality to be fused is determined according to the chain dependency path, the fusion window weight is generated according to the chain propagation time of the chain dependency path, and the classroom behavior data of the modality to be fused is weighted and fused according to the fusion window weight to obtain the fused classroom behavior data. Analyze the classroom behavior fusion data to obtain the interactive behavior scoring index for each target interactive behavior event; The method for collecting the multimodal classroom behavior dataset also includes: Identify the attribute information of current classroom teaching behavior, including classroom type, teaching mode type, and number of students; Based on the attribute information, key temporal features of the classroom are extracted. A first set of multimodal classroom behavior datasets under the key temporal features of the target classroom is obtained at a first data collection frequency. A second set of multimodal classroom behavior datasets not under the key temporal features of the classroom is obtained based on a second data collection frequency. The first data collection frequency is greater than the second data collection frequency. Based on the first set of multimodal classroom behavior datasets and the second set of multimodal classroom behavior datasets, the first set of interactive behavior event encoding sets and the second set of interactive behavior event encoding sets are obtained respectively; After obtaining the first set of interactive event codes and the second set of interactive event codes, the method further includes: Construct a local behavior chain dependency graph model and a global behavior chain dependency graph model according to the first set of interactive behavior event encoding sets and the second set of interactive behavior event encoding sets, respectively. Based on the local behavior chain dependency graph model and the global behavior chain dependency graph model, identify the local chain dependency path and the global chain dependency path based on each target interaction behavior event; According to the local chain dependency path and the global chain dependency path, obtain the local classroom behavior fusion data and the global classroom behavior fusion data; Based on the local classroom behavior fusion data and the global classroom behavior fusion data, the interactive behavior scoring index for each target interactive behavior event is recalculated; The method for constructing the local behavior chain dependency graph model and the global behavior chain dependency graph model respectively includes: The local behavior chain dependency graph model is used to analyze the short-term window behavior triggering relationship of the first set of interactive behavior event encoding sets, and to analyze the edge weight composition of the first set of interactive behavior event encoding sets according to the short-term window behavior triggering relationship. The global behavior chain dependency graph model is used to analyze the long-term window behavior propagation relationship of the second set of interactive behavior event encoding sets, and to analyze the edge weight composition of the second set of interactive behavior event encoding sets according to the long-term window behavior propagation relationship.

2. The classroom teaching behavior analysis method based on multimodal data fusion as described in claim 1, characterized in that, Discretize and encode the interactive behavior events to obtain an interactive behavior event code set. The method includes: The interactive behavior events are obtained by analyzing the feature change rate of the multimodal classroom behavior dataset, comparing the feature change rate with a preset change rate threshold, and dividing the multimodal classroom behavior dataset into temporal behavior segments. The multimodal classroom behavior dataset includes at least visual behavior data, voice behavior data, posture features, and device interaction features. Multidimensional semantic encoding is mapped to the interactive behavior events to construct an interactive behavior event encoding set. The multidimensional semantic encoding includes behavior event type, behavior role, behavior intensity, and behavior duration.

3. The classroom teaching behavior analysis method based on multimodal data fusion as described in claim 2, characterized in that, The behavioral chain dependency graph model identifies the chain dependency path based on each target interaction behavior event. Each graph node in the behavioral chain dependency graph model corresponds to an interaction behavior event, and the edge weight is a weighted result based on the latency decay factor, semantic association factor, role dependency factor, and execution history transfer probability. Traverse the behavior chain dependency graph model to obtain the forward dependency graph nodes with a preset weight for each target interaction behavior event; Linking the nodes of the forward dependency graph based on time constraints yields chain dependency paths based on each target interaction event.

4. The classroom teaching behavior analysis method based on multimodal data fusion as described in claim 1, characterized in that, The method for generating fusion window weights based on the chain propagation time of the chain dependency path includes: Calculate the chain propagation time of each target interaction event on the chain-dependent path to other nodes; A fusion window is generated based on the chain propagation time, and the classroom behavior data of the modal to be fused under the fusion window is obtained. A time decay function is constructed based on the chain propagation time, and a time weight is output based on the time decay function. The time weights are used to fuse the classroom behavior data of the corresponding modalities to be fused under the fusion window.

5. The classroom teaching behavior analysis method based on multimodal data fusion as described in claim 4, characterized in that, The method further includes fusing classroom behavior data of the corresponding modalities to be fused under the fusion window using the time weights mentioned above. Calculate the modal correlation degree between the dataset of classroom behaviors to be integrated and each target interactive behavior event, wherein the modal correlation degree is obtained by calculating the mutual information value; Configure modal weights according to the modal correlation degree, and calculate the joint weight of the modal weights and the time weights; The classroom behavior data of the corresponding modalities to be fused under the fusion window are fused according to the joint weight.

6. The classroom teaching behavior analysis method based on multimodal data fusion as described in claim 1, characterized in that, The method for analyzing the classroom behavior fusion data to obtain the interaction behavior scoring index for each target interaction behavior event includes: The analysis examines the increase rate of interactive event triggers, semantic consistency of interactive events, and fluctuations in the quality of interactive events in the classroom behavior fusion data under the continuous fusion window. The interaction behavior scoring index is obtained by calculating the increase rate of the number of interaction events triggered, the semantic consistency of interaction events, and the fluctuation of interaction event quality.

7. A classroom teaching behavior analysis system based on multimodal data fusion, characterized in that: The steps for implementing the classroom teaching behavior analysis method based on multimodal data fusion as described in any one of claims 1 to 6 include: The event processing module is used to collect a multimodal classroom behavior dataset, identify interactive behavior events in the multimodal classroom behavior dataset, and discretize and encode the interactive behavior events to obtain an interactive behavior event code set. The path identification module is used to establish a behavior chain dependency graph model based on the set of interactive behavior events, and to identify the chain dependency path based on each target interactive behavior event according to the behavior chain dependency graph model. The data fusion acquisition module is used to determine the classroom behavior data of the modality to be fused based on the chain dependency path, generate fusion window weights according to the chain propagation time of the chain dependency path, and perform weighted fusion of the classroom behavior data of the modality to be fused based on the fusion window weights to obtain classroom behavior fusion data; The scoring index acquisition module is used to analyze the classroom behavior fusion data to obtain the interactive behavior scoring index for each target interactive behavior event.

Images