Classroom intelligent analysis method and system driven by multi-modal data analysis large model

Abstract

The invention relates to the technical field of artificial intelligence and intelligent education, in particular to an intelligent classroom analysis method and system driven by a multi-modal data analysis large model, and the method comprises the steps: collecting a multi-modal data flow with a unified timestamp in a classroom scene, and extracting a semantic feature vector in a modal; generating a unified representation vector; semantic conflicts among different modal signals are calculated, and inconsistent feature vectors are generated through coding; evaluating and generating the abstraction degree of the teaching knowledge; calculating and generating a semantic-intention inconsistency index through a preset amplification function; constructing a fusion feature vector; predicting output classroom state probability distribution; triggering a grading intervention strategy in response to the condition that the inconsistency state grade is a polite confusion state; triggering a grading intervention strategy in response to the condition that the probability of the cognitive avalanche state exceeds a preset risk threshold value; an intervention signal is generated and pushed to the teacher end equipment; according to the method, the defect that a traditional model discards conflict information is overcome, and the polite confusion that smile and hesitation coexist can be accurately recognized.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and intelligent education technology, specifically to a classroom intelligent analysis method and system driven by a large multimodal data analysis model. Background Technology

[0002] In modern intelligent teaching environments, real-time and accurate assessment of students’ true cognitive state is crucial for optimizing teaching effectiveness. Traditional classroom analysis methods usually rely on single-modal data or late fusion models. However, when processing multimodal data, these models often discard or ignore intermodal conflict information, making it difficult for them to identify a key teaching state: politeness confusion. In this state, a student's facial expression may show understanding or a smile, but their tone of voice may reveal hesitation, or their textual semantics may express uncertainty. Traditional models cannot effectively capture this inconsistency between semantics and intent. This recognition failure is particularly severe when teaching highly abstract knowledge. Because early politeness confusion cannot be identified, teachers miss the best opportunity to intervene, leading to the continuous accumulation of student confusion, which may evolve into a large-scale cognitive avalanche in the classroom, seriously affecting the quality of teaching. Therefore, how to quantify the semantic conflict between multimodal signals, combine it with the difficulty of the teaching context, accurately identify politeness confusion, and warn of the occurrence of cognitive avalanche is a technical problem that urgently needs to be solved in the field of intelligent teaching. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a classroom intelligent analysis method and system driven by a large multimodal data analysis model. Specifically, the technical solution of this invention is as follows: A multimodal data analysis model-driven intelligent classroom analysis method includes: S1, collect multimodal data streams with a unified timestamp in the classroom scene. The multimodal data streams include: video streams of students' facial expressions; voice waveform streams of students; and classroom text streams. S2, based on multimodal data stream, extracts semantic feature vectors within each modality through a pre-defined single-modal feature extractor; S3 combines the intra-modal semantic feature vectors and projects them onto a unified cross-modal representation space through their respective mapping networks to generate a unified representation vector. S4, based on a unified representation vector, calculates semantic conflicts between signals of different modalities and encodes and generates inconsistent feature vectors; S5 processes the teacher's voice and text stream in parallel and uses a pre-set knowledge graph to evaluate the abstractness of the generated teaching knowledge. S6 combines the inconsistency feature vector with the abstractness of teaching knowledge, and calculates and generates the semantic-intent inconsistency index through a preset amplification function; S7. Compare the semantic-intent inconsistency index with the preset low inconsistency threshold and high inconsistency threshold to determine the current inconsistency state level. S8 aggregates the unified representation vector, the inconsistent feature vector, and the semantic-intent inconsistency index to construct a fused feature vector; S9, input the time series of fused feature vectors into the preset sequence deep learning model, predict the output classroom state probability distribution, which includes the probability of cognitive avalanche state; S10, in response to the inconsistency state level being a polite confusion state, triggers a graded intervention strategy; S11, in response to the probability of a cognitive avalanche state exceeding a preset risk threshold, triggers a tiered intervention strategy; S12 generates and pushes intervention signals to teachers' devices based on a tiered intervention strategy.

[0004] Preferably, a single-modality feature extractor includes: A visual semantic encoder is used to extract visual semantic feature vectors from facial expression video streams; An audio emotion encoder is used to extract audio emotion feature vectors from a speech waveform stream; A text semantic encoder is used to extract text semantic feature vectors from classroom text streams.

[0005] Preferably, the encoding process generates inconsistent feature vectors, including: Based on the unified representation vector, calculate the square of the element-wise difference between the unified representation vectors of different modalities; By combining trainable weight parameters, the squares of element-wise differences are weighted and encoded into an inconsistency feature vector.

[0006] Preferably, the amplification function is a monotonically increasing function; Amplification functions are used to amplify the norm of inconsistent eigenvectors when the abstractness of the teaching knowledge is high, so as to achieve a failure amplification effect.

[0007] Preferably, determining the current level of inconsistency includes: When the semantic-intent inconsistency index is less than or equal to the low inconsistency threshold, it is determined to be in a consistent state. When the semantic-intent inconsistency index is greater than the low inconsistency threshold and less than or equal to the high inconsistency threshold, it is determined to be a politeness confusion state. When the semantic-intent inconsistency index is greater than the high inconsistency threshold, it is determined to be a severe conflict state.

[0008] Preferred, tiered intervention strategies include: In response to an inconsistency level of polite confusion, push a low-priority immediate intervention text signal. In response to the probability of a cognitive avalanche exceeding a preset risk threshold, a high-priority warning signal is pushed out.

[0009] Preferred options also include: Extract the internal hidden state vector of the sequence deep learning model during the prediction process; An unsupervised clustering algorithm is applied to the set of internal latent state vectors to discover emergent classroom states not covered by predefined labels.

[0010] A multimodal data analysis system driven by a large-scale model for intelligent classroom analysis includes: The data acquisition module is used to collect multimodal data streams with a unified timestamp in the classroom setting. The multimodal data streams include: video streams of students' facial expressions; audio waveform streams of students; and classroom text streams. The cross-modal feature extraction module includes: The feature vectorization unit is used to extract semantic feature vectors within a modality based on a multimodal data stream, using a pre-defined single-modal feature extractor. A unified representation unit is used to combine intramodal semantic feature vectors and generate a unified representation vector through their respective mapping networks; The inconsistency coding unit is used to calculate the semantic conflict between different modal signals based on the unified representation vector and to encode and generate inconsistency feature vectors. The quantitative evaluation module includes: The context acquisition unit is used to process the teacher's voice and text stream in parallel and to evaluate the abstraction of the generated teaching knowledge using a pre-defined knowledge graph. The index calculation unit is used to combine the inconsistency feature vector with the abstractness of teaching knowledge, and calculate and generate the semantic-intent inconsistency index through a preset amplification function. The state assessment unit is used to compare the semantic-intent inconsistency index with a preset low inconsistency threshold and a preset high inconsistency threshold to determine the current inconsistency state level. The status recognition module includes: The feature fusion unit is used to aggregate a unified representation vector, inconsistent feature vectors, and semantic-intent inconsistency index to construct a fused feature vector. The time series prediction unit is used to input the time series of fused feature vectors into a preset sequence deep learning model to predict the output classroom state probability distribution, which includes the probability of cognitive avalanche state. The teaching intervention module is used to generate and push graded intervention signals to the teacher's device in response to the inconsistency level meeting preset trigger conditions; the teaching intervention module is also used to generate and push graded intervention signals to the teacher's device in response to the classroom state probability distribution meeting preset trigger conditions.

[0011] Preferably, the status recognition module is also used for: Extract the internal hidden state vector of the sequence deep learning model during the prediction process; An unsupervised clustering algorithm is applied to the set of internal latent state vectors to discover emergent classroom states not covered by predefined labels.

[0012] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention solves the problem of traditional models discarding conflict information by calculating and encoding semantic conflicts between multimodal signals and treating them as independent features. It can accurately identify the polite confusion of students who are both smiling and hesitant. 2. This invention combines the abstractness of teaching knowledge to amplify modal conflicts under highly difficult knowledge, achieving a failure amplification effect. This enables the system to prioritize capturing early confusion under high cognitive load, resulting in more accurate early warnings. 3. This invention explicitly integrates semantics, conflict, and contextual risks, inputs them into a time-series model to provide high-precision early warning of cognitive avalanches, and constructs a real-time teaching regulation closed loop by triggering graded interventions for early confusion and avalanche risk. 4. This invention also enables the automatic discovery of emergent classroom states not covered by predefined labels by performing unsupervised clustering on the internal hidden states of the sequence model, thus giving the system the ability to self-evolve and learn new patterns. Attached Figure Description

[0013] The present invention will be further explained below with reference to the accompanying drawings and embodiments: Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a structural diagram of the system of the present invention. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0015] Example 1: Please see Figure 1 A multimodal data analysis model-driven intelligent classroom analysis method includes: S1, collect multimodal data streams with a unified timestamp in the classroom scene. The multimodal data streams include: video streams of students' facial expressions; voice waveform streams of students; and classroom text streams. S2, based on multimodal data stream, extracts semantic feature vectors within each modality through a pre-defined single-modal feature extractor; S3 combines the intra-modal semantic feature vectors and projects them onto a unified cross-modal representation space through their respective mapping networks to generate a unified representation vector. S4, based on a unified representation vector, calculates semantic conflicts between signals of different modalities and encodes and generates inconsistent feature vectors; S5 processes the teacher's voice and text stream in parallel and uses a pre-set knowledge graph to evaluate the abstractness of the generated teaching knowledge. S6 combines the inconsistency feature vector with the abstractness of teaching knowledge, and calculates and generates the semantic-intent inconsistency index through a preset amplification function; S7. Compare the semantic-intent inconsistency index with the preset low inconsistency threshold and high inconsistency threshold to determine the current inconsistency state level. S8 aggregates the unified representation vector, the inconsistent feature vector, and the semantic-intent inconsistency index to construct a fused feature vector; S9, input the time series of fused feature vectors into the preset sequence deep learning model, predict the output classroom state probability distribution, which includes the probability of cognitive avalanche state; S10, in response to the inconsistency state level being a polite confusion state, triggers a graded intervention strategy; S11, in response to the probability of a cognitive avalanche state exceeding a preset risk threshold, triggers a tiered intervention strategy; S12 generates and pushes intervention signals to teachers' devices based on a tiered intervention strategy.

[0016] This embodiment provides a classroom intelligent analysis method driven by a large multimodal data analysis model; S1, collect multimodal data streams with a unified timestamp in the classroom scene. The multimodal data streams include: video streams of students' facial expressions; voice waveform streams of students; and classroom text streams. The purpose of this step is to obtain the basic data required for analysis. In this embodiment, data is collected in real time by teaching analysis equipment such as high-definition cameras and microphone arrays deployed in the classroom. The key is that all devices are synchronized to a unified high-precision timestamp server to ensure that the collected student facial expression video stream and voice waveform stream have a strict temporal correspondence. At the same time, the system converts the student's voice waveform stream into a classroom text stream in real time through an automatic speech recognition system, and the text stream is also marked with the same timestamp. S2, based on multimodal data stream, extracts semantic feature vectors within each modality through a pre-defined single-modal feature extractor; The purpose of this step is to extract high-dimensional semantic information from the raw data stream. In this embodiment, the system adopts a parallel processing architecture, utilizing three pre-set, single-modal feature extractors trained on large-scale data to process the aligned data stream in S1; for example, a visual semantic encoder. Extracting frown vectors from video streams Audio Emotion Encoder Extracting hesitant pitch vectors from an audio stream Text semantic encoder Extracting the I-uncertain vector from the text stream ; S3 combines the intra-modal semantic feature vectors and projects them onto a unified cross-modal representation space through their respective mapping networks to generate a unified representation vector. The purpose of this step is to solve the challenge of heterogeneous data fusion, namely, the fusion of feature vectors from different modalities such as vision and hearing. Since they have different dimensions and semantic distributions, they cannot be directly compared; therefore, this embodiment constructs a common language space so that the vectors of smiles and hesitant tones can be directly calculated to determine their consistency. In this embodiment, through their respective mapping networks The semantic feature vector from S2 Projecting onto this shared space generates a unified representation vector. The calculation method is as follows: ; in, Refers to modality In time A unified representation vector, for One of them, representing visual, audio, and text respectively; This is the original semantic feature vector of this modality, which originates from the output of step S2; It is a pre-trained modal... Nonlinear transformation networks, such as small multilayer perceptrons; their internal weights It is trained and optimized on a dataset containing cross-modal semantic labels using methods such as contrastive learning; the output of this step They have the same dimensions and similar semantic topology; this makes it possible to calculate the distance or conflict between them, which is the core premise for the subsequent S4 calculation of inconsistency features. S4, based on a unified representation vector, calculates semantic conflicts between signals of different modalities and encodes and generates inconsistent feature vectors; The purpose of this step is to address the shortcoming of traditional late fusion models that discard intermodal conflict information; the motivation of this approach is to not discard the conflict, but to quantify and encode the conflict itself as a new, atomic, independent feature. In this embodiment, based on the unified representation vector generated by S3, the calculation is performed at the same time point. The degree of semantic conflict is encoded as an inconsistency feature vector. The calculation method is as follows: ; in, It is time Inconsistent feature vectors; and It is the unified representation vector of the two different modes output by S3; It is the square of their element-wise differences in the shared space; it is a vector representing their semantic differences or conflicts in various dimensions. It is a trainable weight parameter that represents the modality. and The importance of the conflict arises from the optimization performed via backpropagation during the training of a large model, with the data source consisting of samples containing inconsistent labels; this vector The norm size represents the overall degree of semantic-intention inconsistency; for example, when smiling... and hesitant tone When they occur simultaneously, the square of their vector difference in the same space will be very large, leading to With high energy, the key event of politeness confusion was captured atomistically; the output It will serve as the core input for subsequent S6 and S8; S5 processes the teacher's voice and text stream in parallel and uses a pre-set knowledge graph to evaluate the abstractness of the generated teaching knowledge. The purpose of this step is to obtain the contextual difficulty of the teaching; in this embodiment, the system processes the teacher's voice and text stream in parallel. That is, the content taught in class, utilizing a pre-set knowledge graph or large language model. Assess the current moment The level of abstraction of the content being taught; Generate teaching knowledge abstraction level It refers to the cognitive complexity of the current teaching content; in this embodiment, it is a scalar, for example... ; where 1.0 represents everyday greetings, and 3.0 represents high-density abstract knowledge such as quantum superposition states; this parameter will be used for the calculation of S6; S6 combines the inconsistency feature vector with the abstractness of teaching knowledge, and calculates and generates the semantic-intent inconsistency index through a preset amplification function; This step aims to quantify the failure amplification effect; its motivation is that the same degree of smile plus hesitation is caused by… Measurement in teaching high-density abstract knowledge The risk of confusion is much higher than in casual conversation; In this embodiment, combined with the output of S4 and S5 output Calculate the semantic-intent inconsistency index at the current time t. Semantic-IntentDiscrepancyIndex (SIDI) is calculated as follows: ; in, It is the semantic-intent inconsistency index, which is a scalar. It is the inconsistency feature vector output by S4; yes p-norm, for example That is, the Euclidean length, which represents the original collision amplitude of the student's multimodal signal, is a scalar; The level of knowledge abstraction output by S5 is a scalar. It is a pre-defined monotonically increasing amplification function; its form is, for example, as follows: ,in It is the amplification factor, obtained by calibrating on data from courses of varying difficulty; when At a very high level, The function will be significantly amplified. The value of leads to A surge; this allows the system to accurately capture serious politeness confusion that occurs during challenging instruction; the output This will serve as the input for subsequent S7 and S8; S7. Compare the semantic-intent inconsistency index with the preset low inconsistency threshold and high inconsistency threshold to determine the current inconsistency state level. The purpose of this step is to perform a graded evaluation of the SIDI index calculated by S6. Low inconsistency threshold and high inconsistency threshold This refers to the boundary value used to delineate the degree of inconsistency; its source is: statistically calibrated on a benchmark dataset containing expert-labeled politeness confusion and genuine confusion, for example, by taking the SIDI index. The 20% and 80% quantiles of the distribution were obtained; Used to filter out normal intermodal noise. Used to distinguish between moderate inconsistency and severe conflict; The comparison logic will be detailed in Example 5, for example: For consistency; Confused by politeness; This indicates a severe conflict; the inconsistency level of this output will be used as input to S10. S8 aggregates the unified representation vector, the inconsistent feature vector, and the semantic-intent inconsistency index to construct a fused feature vector; This step is crucial for achieving early integration; traditional methods only integrate... This solution will... Already in a unified representation space Calculated in the same space and context scalar Explicit concatenation enables downstream models to simultaneously perceive semantic, conflict, and contextual risks; in this embodiment, a fused feature vector is constructed. The calculation method is as follows: ; in, It is a unified fused feature vector; it represents a vector concatenation operation. The source is step S3; The source is step S4; The source is step S6, where it is added as a scalar or one-dimensional feature to the concatenation; this step generates a feature vector with high information density and a consistent feature space. This vector will be used as the input to S9 for the time-series state evolution model; S9, input the time series of fused feature vectors into the preset sequence deep learning model, predict the output classroom state probability distribution, which includes the probability of cognitive avalanche state; This step utilizes a time series model to capture the temporal dependencies of classroom status; In this embodiment, time series Input into a pre-defined sequence deep learning model For example, in long short-term memory networks or transformer models; The model's parameters were obtained through supervised training on labeled sequence data of classroom states, and its prediction method is as follows: ; in, It is a probability distribution of classroom states, which is a probability vector; its dimensions correspond to predefined classroom states, such as focused, confused, and active, and explicitly include cognitive avalanche states; It is a pre-defined sequence model; It is the time series of the fused feature vectors output by S8; because The middle explicitly includes Conflict and Contextual risk, the model It can not only identify traditional states, but also identify complex states based on inconsistencies with high accuracy, especially cognitive avalanches; the output... The probability of the cognitive avalanche state will be used as the input of S11; S10, in response to the inconsistency state level being a polite confusion state, triggers a graded intervention strategy; S11, in response to the probability of a cognitive avalanche state exceeding a preset risk threshold, triggers a tiered intervention strategy; S12, based on the tiered intervention strategy, generates and pushes intervention signals to the teacher's terminal device; These three steps together constitute a closed loop for real-time teaching intervention; Define the risk threshold in S11 It refers to the probability threshold used to judge that a cognitive avalanche is about to occur; its source is: calibration by analyzing the receiver operation characteristic curve of the cognitive avalanche state in historical data, in order to balance the accuracy and recall of the warning. The system monitors the outputs of S7 and S9 in real time and identifies two triggering conditions: When the inconsistency state level output by S7 is within time S10 is triggered when the state changes to a polite confusion state, i.e., state 1. When S9 outputs The probability of a cognitive avalanche state in the middle exceeds the preset risk threshold. S11 is triggered at this time; S12 is implemented based on a tiered intervention strategy: In response to S10, the system determines that politeness confusion has occurred; it immediately pushes a low-priority instant intervention text signal to the teacher's display device, such as a teleprompter or tablet; example of the signal content: a high SIDI event has been detected; suggestion: restate the abstract concept just mentioned, or ask questions to students with inconsistent expressions; In response to S11, the system determines that a cognitive avalanche is about to occur; it immediately pushes a high-priority warning signal; example of the signal content: Warning: A large-scale cognitive collapse risk has been detected; Recommendation: Immediately suspend teaching and conduct a classroom quiz or check. The method described in this embodiment innovatively designs the S4 inconsistency feature vector. S6 Semantic-Intent Inconsistency Index It successfully transforms the intermodal conflict information discarded in traditional technologies into quantifiable and usable core features; it solves the problem that politeness confusion is difficult to recognize by traditional models because of the inconsistency between its multimodal signals such as smiling and hesitation. Furthermore, through S8 and Integration Combined with the time-series model of S9, it achieved high-precision early warning for complex states such as cognitive avalanche; finally, through the graded intervention of S10-S12, the core finding of inconsistency was transformed from passive post-event diagnosis to real-time intervention, and a closed-loop control system was constructed that dynamically adjusts teaching strategies based on students' true cognitive state rather than surface semantics, which significantly improved the effectiveness and pertinence of teaching.

[0017] Example 2: Single-modal feature extractors include: A visual semantic encoder is used to extract visual semantic feature vectors from facial expression video streams; An audio emotion encoder is used to extract audio emotion feature vectors from a speech waveform stream; A text semantic encoder is used to extract text semantic feature vectors from classroom text streams.

[0018] This embodiment is a specific implementation of the single-modality feature extractor preset in step S2 of embodiment 1; In this embodiment, the single-modality feature extractor includes: Visual semantic encoder Its purpose is to extract high-level visual semantics from facial expression video streams; in this embodiment, it is a pre-trained deep convolutional neural network used to extract visual semantic feature vectors from facial expression video streams. ;For example, It is a vector representation corresponding to a smile or a frown; Audio Emotion Encoder Its purpose is to extract emotion and intonation information from speech waveform streams; in this embodiment, it is a pre-trained audio processing model used to extract audio emotion feature vectors from speech waveform streams. ;For example, It is a vector representation corresponding to hesitant tone or confident tone; Text Semantic Encoder Its purpose is to extract semantic content from classroom text streams; in this embodiment, it is a pre-trained large-scale language model used to extract text semantic feature vectors from classroom text streams. ;For example, It corresponds to the vector representation that I understand or that I am unsure about; By employing dedicated deep encoders for different modalities of vision, audio, and text, this embodiment achieves deep semantic extraction of the raw data stream, rather than remaining at surface physical features such as pitch and pixel values. This provides high-quality, high-dimensional, and high-semantic-fidelity feature inputs for the subsequent unified representation mapping in S3 and cross-modal conflict calculation in S4, greatly improving the performance of inconsistency feature vectors. The accuracy and reliability of [the system / mechanism].

[0019] Example 3: Encoding generates inconsistent feature vectors, including: Based on the unified representation vector, calculate the square of the element-wise difference between the unified representation vectors of different modalities; By combining trainable weight parameters, the squares of element-wise differences are weighted and encoded into an inconsistency feature vector.

[0020] This embodiment is a concretization of the encoding and generation of inconsistent feature vectors in step S4 of embodiment 1; In this embodiment, the encoding generates inconsistent feature vectors. The process specifically includes: Based on the unified representation vector, the square of the element-wise difference between the unified representation vectors of different modalities is calculated; that is, the square of the element-wise difference is calculated. ,in This computation is performed in the shared space constructed by S3, and the result is a vector representing the modes. and The magnitude of differences or conflicts across various semantic dimensions; Combined with trainable weight parameters The squares of the element-wise differences are weighted and encoded into an inconsistency feature vector; that is, the following steps are performed. The operation; the trainable weight parameters This refers to the different modal pairs, such as visual-auditory pairs, that the model learns autonomously. Auditory-text pairs The importance of the conflict between them; its source is optimized through backpropagation training on samples containing inconsistent labels; The weights can be trained using the sum of squared differences element-wise. This embodiment achieves refined encoding of conflict information through a combined approach; the square of the element-wise difference preserves the specific semantic dimension information of the conflict occurrence, rather than just a scalar distance; it is trainable. Weights enable the model to autonomously learn, through a data-driven approach, which modal conflicts, such as visual-auditory conflicts, exist. Compared to other conflicts such as auditory-text conflicts It is better able to predict key teaching states such as politeness confusion; this makes the final output... The vectors are more directional and contain more key information needed for SIDI calculation, thus improving the accuracy of S6 evaluation.

[0021] Example 4: The amplification function is a monotonically increasing function; Amplification functions are used to amplify the norm of inconsistent eigenvectors when the abstractness of the teaching knowledge is high, so as to achieve a failure amplification effect.

[0022] This embodiment is a specific implementation of the amplification function preset in step S6 of embodiment 1; In this embodiment, the amplification function It is a monotonically increasing function; the technical motivation is that the higher the abstraction of the taught knowledge, the greater the risk of teaching failure due to students' politeness confusion; therefore, the function... Must ensure The larger the value, the greater the amplification factor of its output value; in this embodiment, It can be an exponential function, for example ,in It is an adjustable amplification factor; This amplification function is used to adjust the level of abstraction of teaching knowledge. When the norm is high, it amplifies the inconsistency eigenvector norm. This is done to achieve a failure amplification effect; when teachers are teaching high-density abstract knowledge, for example... , It will get a larger value; conversely, in casual conversation, for example... , It does not amplify the effect; By The SIDI index is explicitly designed as a monotonically increasing function, such as an exponential function. In this embodiment, the SIDI index is made to... Become highly sensitive to the level of abstraction of knowledge in the teaching context; in teaching high-difficulty topics At high levels, even minor primal conflicts Even smaller ones will be Significant magnification, leading to Surge and may cross Thresholds enable the system to prioritize capturing politeness confusion that occurs under high cognitive load and is most likely to lead to teaching failure, while automatically filtering out meaningless modal conflicts that occur during small talk, greatly improving the targeting and effectiveness of early warnings.

[0023] Example 5: Determine the current level of inconsistency, including: When the semantic-intent inconsistency index is less than or equal to the low inconsistency threshold, it is determined to be in a consistent state. When the semantic-intent inconsistency index is greater than the low inconsistency threshold and less than or equal to the high inconsistency threshold, it is determined to be a politeness confusion state. When the semantic-intent inconsistency index is greater than the high inconsistency threshold, it is determined to be a severe conflict state.

[0024] This embodiment is a concretization of step S7 in embodiment 1, which determines the current level of inconsistency. In this embodiment, this step utilizes the low inconsistency threshold defined in S7. and high inconsistency threshold The SIDI index calculated for S6 The comparison is performed using the following logic: When the semantic-intent inconsistency index Less than or equal to the low inconsistency threshold The state is determined to be consistent, state 0; this means that the signals between modes are consistent, or the conflict amplitude is within an acceptable noise range, and the student's state, such as focused or distracted, is credible. When the semantic-intent inconsistency index Greater than And less than or equal to the high inconsistency threshold The state is identified as politeness confusion state 1; this indicates that the system has detected a key semantic-intent inconsistency, and the student may be hiding confusion; this is the core trigger condition for S10. When the semantic-intent inconsistency index Greater than The condition is identified as a severe conflict state 2; this indicates an extreme conflict between modes, which may indicate other abnormal states such as mockery or equipment failure, requiring further analysis. By setting and With dual thresholds, this embodiment achieves the control of the SIDI index. The evaluation results are finely graded; it can not only distinguish between consistent state 0 and inconsistent state 1 or 2, but also further distinguish between polite confusion state 1 and serious conflict state 2, which are of primary concern to the system. This three-level state division provides a clear and operable decision basis for implementing graded intervention strategies with different priorities in S10 and S12, avoiding the inefficiency of a one-size-fits-all approach to intervention.

[0025] Example 6: Tiered intervention strategies include: In response to an inconsistency level of polite confusion, push a low-priority immediate intervention text signal. In response to the probability of a cognitive avalanche exceeding a preset risk threshold, a high-priority warning signal is pushed out.

[0026] This embodiment is a concretization of the graded intervention strategies S10, S11 and S12 in Embodiment 1; In this embodiment, the tiered intervention strategy executes different closed-loop intervention actions based on the trigger levels activated in S10 and S11: In response to an inconsistency state level of polite confusion (Level 1 trigger), a low-priority immediate intervention text signal is pushed; the trigger source is S10, i.e., S7 decision. In state 1, the action is as follows: the system determines that politeness confusion has occurred; it immediately pushes a low-priority instant intervention text signal to the teacher's display device, such as a teleprompter; the purpose is to correct the individual's early confusion in real time and prevent it from evolving into a cognitive avalanche. The probability of responding to a cognitive avalanche exceeds a preset risk threshold. This refers to a secondary trigger that pushes a high-priority warning signal; its trigger source is S11, which is determined by S9 to trigger a cognitive avalanche. Its actions are as follows: the system determines that a cognitive avalanche is about to occur or is already occurring; it immediately pushes a high-priority warning signal, such as a pop-up window or sound prompt; its purpose is to take emergency measures to stop the overall and serious risk of cognitive collapse in the classroom, and to advise the teacher to immediately suspend teaching. By distinguishing between Level 1 SIDI events and Level 2 cognitive avalanche warning triggers, and matching intervention signals of different priorities, this embodiment makes teaching intervention more targeted and timely. Low-priority signals are used to immediately correct individual politeness confusion and prevent it from escalating, with minimal interference to the teaching process. High-priority signals are used for emergency braking to prevent cognitive avalanche in the classroom as a whole, with high intervention intensity. This hierarchical mechanism ensures the accuracy of intervention, which can address early problems in a timely manner and avoid unnecessary teaching interruptions caused by over-intervention, significantly optimizing the efficiency of the teaching loop and user experience.

[0027] Example 7: This method also includes: Extract the internal hidden state vector of the sequence deep learning model during the prediction process; An unsupervised clustering algorithm is applied to the set of internal latent state vectors to discover emergent classroom states not covered by predefined labels.

[0028] This embodiment is a further optimization based on the method described in Embodiment 1, by adding an unsupervised discovery step; In this embodiment, the method further includes: Extracting Sequence Deep Learning Model Internal hidden state vector in the prediction process The internal hidden state vector This refers to the sequence model in S9. For example, Long Short-Term Memory networks or transformer models at processing time points Input The hidden state representation generated in the last time step; yes A highly condensed summary of the entire time series, its source is Calculation of intermediate products; For the internal hidden state vector The dataset is subjected to unsupervised clustering to discover emerging classroom states not covered by predefined labels; in the model After training, it was applied to a large amount of classroom data and a massive amount of data was collected. Vectors; and these Vectors are applied using an unsupervised clustering algorithm; dense clusters identified by the clustering algorithm that are not covered by predefined labels in S9, such as focus and confusion, represent emerging classroom states; for example, the system may automatically discover a cluster whose characteristics are... A slight frown, Prolonged silence, Moderate, which might be labeled as critical thinking by experts; Through the Hidden state This embodiment achieves automatic discovery of new teaching models through unsupervised clustering; because Condensed The information, and It also includes the original features of this invention. Inconsistency and Contextual risk features enable clustering algorithms to leverage these new characteristics to discover valuable teaching patterns, such as critical thinking, that are unrecognizable by traditional methods. This allows the system not only to predict known classroom conditions but also to learn and evolve autonomously, greatly expanding the system's analytical capabilities and cognitive boundaries.

[0029] Example 8: Please see Figure 2 A multimodal data analysis system driven by a large model for intelligent classroom analysis includes: The data acquisition module is used to collect multimodal data streams with a unified timestamp in the classroom setting. The multimodal data streams include: video streams of students' facial expressions; audio waveform streams of students; and classroom text streams. The cross-modal feature extraction module includes: The feature vectorization unit is used to extract semantic feature vectors within a modality based on a multimodal data stream, using a pre-defined single-modal feature extractor. A unified representation unit is used to combine intramodal semantic feature vectors and generate a unified representation vector through their respective mapping networks; The inconsistency coding unit is used to calculate the semantic conflict between different modal signals based on the unified representation vector and to encode and generate inconsistency feature vectors. The quantitative evaluation module includes: The context acquisition unit is used to process the teacher's voice and text stream in parallel and to evaluate the abstraction of the generated teaching knowledge using a pre-defined knowledge graph. The index calculation unit is used to combine the inconsistency feature vector with the abstractness of teaching knowledge, and calculate and generate the semantic-intent inconsistency index through a preset amplification function. The state assessment unit is used to compare the semantic-intent inconsistency index with a preset low inconsistency threshold and a preset high inconsistency threshold to determine the current inconsistency state level. The status recognition module includes: The feature fusion unit is used to aggregate a unified representation vector, inconsistent feature vectors, and semantic-intent inconsistency index to construct a fused feature vector. The time series prediction unit is used to input the time series of fused feature vectors into a preset sequence deep learning model to predict the output classroom state probability distribution, which includes the probability of cognitive avalanche state. The teaching intervention module is used to generate and push graded intervention signals to the teacher's device in response to the inconsistency level meeting preset trigger conditions; the teaching intervention module is also used to generate and push graded intervention signals to the teacher's device in response to the classroom state probability distribution meeting preset trigger conditions.

[0030] This embodiment provides a classroom intelligent analysis system driven by a large multimodal data analysis model. This system is a combination of hardware and software, used to execute the method described in any one of embodiments 1-7; the system includes: The data acquisition module is designed to perform step S1. In this embodiment, the module physically includes a high-definition camera, a microphone array, and a high-precision timestamp synchronization server. The module is used to acquire video streams of students' facial expressions and audio waveforms with uniform timestamps in a classroom setting, and to generate classroom text streams through an internal automatic speech recognition engine. The cross-modal feature extraction module is designed to perform steps S2, S3, and S4; this module includes: The feature vectorization unit is used to perform step S2; this unit contains a pre-defined single-modality feature extractor, such as... Used to extract semantic feature vectors within each modality. ; A unified representation unit is used to perform the S3 step; each unit contains its own mapping network. , used to Projecting onto a unified cross-modal representation space generates a unified representation vector. ; Inconsistency coding unit, used to perform step S4; this unit is used to base on and and utilize trainable weights Calculate and encode inconsistent feature vectors ; The quantitative evaluation module aims to execute steps S5, S6, and S7; this module includes: The context acquisition unit is used to execute step S5; this unit contains a pre-defined knowledge graph. It is used for parallel processing of the teacher's speech-text stream to evaluate the level of abstraction of the generated instructional knowledge. ; The exponent calculation unit is used to execute step S6; this unit contains a preset amplification function. , used for combination and Calculate the semantic-intent inconsistency index. ; The state evaluation unit is used to perform step S7; this unit stores a preset low inconsistency threshold. and high inconsistency threshold , used to In comparison, determine the current level of inconsistency. The state recognition module is designed to execute steps S8 and S9; this module includes: The feature fusion unit performs the S8 step; this unit is used for aggregation. , as well as Concatenate to construct a fused feature vector ; The time series prediction unit is used to execute step S9; this unit contains a pre-defined sequence deep learning model. Used to receive Time series, predict the probability distribution of classroom state. ,Should Including the probability of a cognitive avalanche state; The teaching intervention module aims to execute steps S10, S11, and S12; this module is a closed-loop control actuator; it is used for: The inconsistency level output by the quantitative evaluation module meets the preset triggering conditions, such as state 1 polite confusion. The classroom state probability distribution output by the state recognition module meets preset triggering conditions, such as cognitive avalanche. ; Based on the above triggering conditions, tiered intervention signals are generated and pushed to the teacher's terminal device, such as the teacher's tablet or teleprompter. The system in this embodiment provides a complete technical architecture from data acquisition, multimodal inconsistency quantification, contextual risk assessment, temporal state prediction to closed-loop intervention through the collaborative work of the above modules. The system can automatically and in real time execute the methods described in Embodiment 1, transforming abstract teaching theories into deployable hardware and software entities, realizing intelligent and refined management and intervention of the classroom teaching process, and effectively improving teaching quality and efficiency.

[0031] Example 9: The status recognition module is also used for: Extract the internal hidden state vector of the sequence deep learning model during the prediction process; An unsupervised clustering algorithm is applied to the set of internal latent state vectors to discover emergent classroom states not covered by predefined labels.

[0032] This embodiment is a functional extension of the state recognition module in embodiment 8, corresponding to the method described in embodiment 7; In this embodiment, the state recognition module is also used for: Extracting Sequence Deep Learning Model Internal hidden state vector in the prediction process This function is performed by the execution unit of the timing prediction unit S9 during calculation. At the same time, synchronize its internal Vector export; For the internal hidden state vector The system applies an unsupervised clustering algorithm to discover emerging classroom states not covered by predefined labels; the state recognition module also includes an unsupervised learning engine that periodically collects... The system collects and performs clustering, feeding the newly discovered state clusters back to the system knowledge base; By adding this unsupervised clustering function to the state recognition module, this embodiment enables the system to have the ability to self-evolve; the system is no longer limited to performing preset diagnostic tasks such as detecting confusion or cognitive avalanche, but can continuously mine massive amounts of classroom data during operation and utilize... and With its deep features, the system can autonomously discover new and valuable classroom models; this allows the system's knowledge base and intervention strategies to be continuously enriched and improved, maintaining the advanced nature and adaptability of the technology.

[0033] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A classroom intelligent analysis method driven by a large multimodal data analysis model, characterized in that, include: S1, collect multimodal data streams with a unified timestamp in the classroom scene. The multimodal data streams include: student facial expression video stream; student speech waveform stream; and classroom text stream; S2, based on multimodal data stream, extracts semantic feature vectors within each modality through a pre-defined single-modal feature extractor; S3 combines the intra-modal semantic feature vectors and projects them onto a unified cross-modal representation space through their respective mapping networks to generate a unified representation vector. S4, based on a unified representation vector, calculates semantic conflicts between signals of different modalities and encodes and generates inconsistent feature vectors; S5 processes the teacher's voice and text stream in parallel and uses a pre-set knowledge graph to evaluate the abstractness of the generated teaching knowledge. S6 combines the inconsistency feature vector with the abstractness of teaching knowledge, and calculates and generates the semantic-intent inconsistency index through a preset amplification function; S7. Compare the semantic-intent inconsistency index with the preset low inconsistency threshold and high inconsistency threshold to determine the current inconsistency state level. S8 aggregates the unified representation vector, the inconsistent feature vector, and the semantic-intent inconsistency index to construct a fused feature vector; S9, input the time series of fused feature vectors into the preset sequence deep learning model, predict the output classroom state probability distribution, which includes the probability of cognitive avalanche state; S10, in response to the inconsistency state level being a polite confusion state, triggers a graded intervention strategy; S11, in response to the probability of a cognitive avalanche state exceeding a preset risk threshold, triggers a tiered intervention strategy; S12 generates and pushes intervention signals to teachers' devices based on a tiered intervention strategy.

2. The classroom intelligent analysis method driven by a large multimodal data analysis model according to claim 1, characterized in that, Single-modal feature extractors include: A visual semantic encoder is used to extract visual semantic feature vectors from facial expression video streams; An audio emotion encoder is used to extract audio emotion feature vectors from a speech waveform stream; A text semantic encoder is used to extract text semantic feature vectors from classroom text streams.

3. The classroom intelligent analysis method driven by a large multimodal data analysis model according to claim 1, characterized in that, Encoding generates inconsistent feature vectors, including: Based on the unified representation vector, calculate the square of the element-wise difference between the unified representation vectors of different modalities; By combining trainable weight parameters, the squares of element-wise differences are weighted and encoded into an inconsistency feature vector.

4. The classroom intelligent analysis method driven by a large multimodal data analysis model according to claim 1, characterized in that, The amplification function is a monotonically increasing function; Amplification functions are used to amplify the norm of inconsistent eigenvectors when the abstractness of the teaching knowledge is high, so as to achieve a failure amplification effect.

5. The classroom intelligent analysis method driven by a large multimodal data analysis model according to claim 1, characterized in that, Determine the current level of inconsistency, including: When the semantic-intent inconsistency index is less than or equal to the low inconsistency threshold, it is determined to be in a consistent state. When the semantic-intent inconsistency index is greater than the low inconsistency threshold and less than or equal to the high inconsistency threshold, it is determined to be a politeness confusion state. When the semantic-intent inconsistency index is greater than the high inconsistency threshold, it is determined to be a severe conflict state.

6. The classroom intelligent analysis method driven by a large multimodal data analysis model according to claim 1, characterized in that, Tiered intervention strategies include: In response to an inconsistency level of polite confusion, push a low-priority immediate intervention text signal; In response to the probability of a cognitive avalanche exceeding a preset risk threshold, a high-priority warning signal is pushed out.

7. The classroom intelligent analysis method driven by a large multimodal data analysis model according to claim 1, characterized in that, Also includes: Extract the internal hidden state vector of the sequence deep learning model during the prediction process; An unsupervised clustering algorithm is applied to the set of internal latent state vectors to discover emergent classroom states not covered by predefined labels.

8. A classroom intelligent analysis system driven by a large multimodal data analysis model, applied to the classroom intelligent analysis method driven by a large multimodal data analysis model as described in any one of claims 1-7, characterized in that, include: The data acquisition module is used to collect multimodal data streams with a unified timestamp in the classroom setting. The multimodal data streams include: video streams of students' facial expressions; audio waveform streams of students; and classroom text streams. The cross-modal feature extraction module includes: The feature vectorization unit is used to extract semantic feature vectors within a modality based on a multimodal data stream, using a pre-defined single-modal feature extractor. A unified representation unit is used to combine intramodal semantic feature vectors and generate a unified representation vector through their respective mapping networks; The inconsistency coding unit is used to calculate the semantic conflict between different modal signals based on the unified representation vector and to encode and generate inconsistency feature vectors. The quantitative evaluation module includes: The context acquisition unit is used to process the teacher's voice and text stream in parallel and to evaluate the abstraction of the generated teaching knowledge using a pre-defined knowledge graph. The index calculation unit is used to combine the inconsistency feature vector with the abstractness of teaching knowledge, and calculate and generate the semantic-intent inconsistency index through a preset amplification function. The state assessment unit is used to compare the semantic-intent inconsistency index with a preset low inconsistency threshold and a preset high inconsistency threshold to determine the current inconsistency state level. The status recognition module includes: The feature fusion unit is used to aggregate a unified representation vector, inconsistent feature vectors, and semantic-intent inconsistency index to construct a fused feature vector. The time series prediction unit is used to input the time series of fused feature vectors into a preset sequence deep learning model to predict the output classroom state probability distribution, which includes the probability of cognitive avalanche state. The teaching intervention module is used to generate and push graded intervention signals to the teacher's device in response to the inconsistency level meeting preset trigger conditions; the teaching intervention module is also used to generate and push graded intervention signals to the teacher's device in response to the classroom state probability distribution meeting preset trigger conditions.

9. A classroom intelligent analysis system driven by a large multimodal data analysis model according to claim 8, characterized in that, The status recognition module is also used for: Extract the internal hidden state vector of the sequence deep learning model during the prediction process; An unsupervised clustering algorithm is applied to the set of internal latent state vectors to discover emergent classroom states not covered by predefined labels.

Images