Smart classroom adaptability regulation and control method based on multi-mode and large language model

Abstract

The invention belongs to the technical field of artificial intelligence, and particularly discloses a smart classroom adaptability regulation and control method based on multiple modes and a large language model, and the method comprises the steps: collecting multi-source data of a smart classroom environment; performing data preprocessing on the multi-source data, extracting key features of each mode, and constructing a feature library of environment features and behavior features; fusing the environment features and the behavior features by using an orthogonal sequence fusion strategy, and constructing a fusion feature vector; inputting the fusion feature vector into a large language model to obtain a judgment result of the classroom state, and generating an executable regulation and control instruction according to the judgment result; the regulation and control instruction is transmitted to a central control system of the smart classroom, so that real-time regulation and control of the environment and interactive reminding are realized; environment feedback data and student behavior feedback data after regulation and control are collected, and a regulation and control strategy is updated through a reinforcement learning algorithm based on the feedback data. According to the invention, comprehensive performance, simplicity, convenience and real-time performance of adaptive regulation and control can be realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence, and more particularly, relates to a wisdom classroom appropriateness regulation method based on multi-modal and large language model. BACKGROUND

[0002] With the continuous development of wisdom education, the deep integration of information technology and classroom teaching gradually forms a new teaching environment, namely, an intelligent classroom (IC). In the IC scenario, the learning state of students is not only related to the teaching content, but also influenced by environmental factors (such as light, noise, temperature and humidity, carbon dioxide concentration, etc.) and student behavior performance (such as posture, gaze, facial expression, etc.). Therefore, how to identify the classroom appropriateness of students in real time and generate effective regulation suggestions in a complex and dynamic classroom environment has become a key problem in wisdom education research.

[0003] Existing classroom behavior analysis methods mostly rely on single modality (Single Modality) data. For example, posture estimation algorithms based on computer vision judge the attention level of students by extracting human skeleton key points, but in the classroom scenario, the recognition accuracy is significantly reduced due to insufficient light, frequent occlusion, or high personnel density. Classroom research methods based on speech analysis mainly focus on the language interaction of students or teachers, and the typical representatives are Flanders Interaction Analysis System (FIAS) and improved i-FIAS. However, FIAS requires manual observation and coding, which is low in efficiency and strong in subjectivity; i-FIAS has improved the analysis of language interaction, but it lacks coverage of non-verbal behaviors (such as body movements and expression changes). In addition, data analysis based on environmental sensors can capture environmental conditions such as light intensity, noise level, temperature and humidity, but this kind of method is usually used independently and cannot form joint modeling with student behavior information.

[0004] With the development of multi-modal fusion (Multi-modal Fusion, MMF) technology, researchers have tried to fuse environmental modalities and behavior modalities to improve the robustness of classroom state recognition. Common fusion methods include early fusion, feature layer fusion and late fusion. However, traditional data splicing or result weighting methods are prone to information redundancy or modality conflict when dealing with multi-source heterogeneous data, making it difficult to ensure the explainability and stability of the fusion results. The orthogonal sequential fusion (Orthogonal Sequential Fusion, OSF) method proposed in recent years can effectively alleviate the problem of information redundancy by introducing orthogonal constraints to gradually fuse different modalities, and has better adaptability and explainability, which is suitable for complex scenarios such as wisdom classrooms.

[0005] On the other hand, large language models (LLMs) have shown strong generation and reasoning capabilities in the field of natural language processing (NLP). Through low-rank adaptation (LoRA), quantization, distillation, and pruning, LLMs can be deployed efficiently while maintaining performance. In the field of smart education, some studies have used LLMs to analyze teaching texts or speech, but most methods have remained at the level of single-modal result interpretation, lacking deep fusion of multi-source heterogeneous data and real-time regulation of classroom status.

[0006] In summary, the existing classroom learning state recognition and regulation technology generally has the following problems: first, the data modality is single, making it difficult to fully reflect the student learning state; second, the fusion method is simple, making it difficult to effectively handle conflicts and redundancies between multi-source heterogeneous data; third, intelligent models are mostly focused on state recognition, lacking real-time and personalized regulation suggestions for teaching practice.

[0007] Therefore, how to solve the defects of one-sided information, complex processing, and feedback lag in the traditional classroom state recognition process is a technical problem that needs to be solved at present. SUMMARY

[0008] In view of the defects of the prior art, the purpose of the present application is to solve the defects of one-sided information, complex processing, and feedback lag in the traditional classroom state recognition process.

[0009] To achieve the above purpose, in a first aspect, the present application provides a smart classroom appropriateness regulation method based on multi-modal and large language models, comprising: Collecting multi-source data of the classroom environment of the smart classroom, the multi-source data including environmental modal data and behavioral modal data; Data preprocessing of the environmental modal data and the behavioral modal data, extracting key features of each modality, and constructing a feature library of environmental features and behavioral features; Fusing the environmental features and behavioral features using an orthogonal sequential fusion strategy to construct a fusion feature vector; Inputting the fusion feature vector into a large language model to obtain a discrimination result of the classroom state, and generating an executable regulation instruction according to the discrimination result; Delivering the regulation instruction to the central control system of the smart classroom to realize real-time regulation of environmental and interactive reminders; Collecting feedback data of the regulated environment and student behavior, and updating the regulation strategy based on the feedback data through a reinforcement learning algorithm to realize dynamic closed-loop optimization of appropriateness recognition and regulation.

[0010] Optionally, the multi-source data of the smart classroom classroom environment includes: The temperature, humidity, light intensity, carbon dioxide concentration and noise level data are collected by the environmental data collection unit as the environmental modal data; The behavior visual collection unit captures student posture, gaze direction, facial expression and speaking frequency through image recognition algorithm as behavior modal data; A multi-thread synchronization mechanism is used to ensure the time consistency of the environmental modal data and the behavior modal data, and to realize the synchronization of multi-modal collection.

[0011] Optionally, the data preprocessing process includes: The sliding window-based outlier detection algorithm is used to eliminate sensor drift and short-term noise in the multi-source data; The time series alignment algorithm based on dynamic time warping is used to realize the time synchronization of the multi-source data; The Z-Score standardization strategy is used to normalize the multi-source data to unify the feature scale.

[0012] Optionally, the process of obtaining the fusion feature vector includes: The environmental features and behavior features are mapped to subspaces that are not correlated with each other; The environmental features and behavior features are sequentially fused based on time sequence priority to obtain a preliminary fusion result; The fusion result is optimized through an attention weighting mechanism to obtain the final fusion feature vector.

[0013] Optionally, the control instruction includes at least one of: The environmental control instruction is used to adjust the light, air quality, temperature and humidity, and ventilation system; The interactive prompt instruction is used to remind teachers or students to adjust their posture, focus their attention, or promote teacher-student interaction; The adaptive feedback instruction is used to guide the system to adjust the sampling frequency or control amplitude to optimize energy consumption and response time delay.

[0014] Optionally, the large language model is constructed based on Llama 8B; the large language model is lightened by using low-rank adaptation technology, and the model size is reduced while maintaining the accuracy level through quantization acceleration mechanism.

[0015] Optionally, the method further includes: Calculate the appropriateness index of the student's classroom, including attention concentration, emotional stability, participation and environmental comfort; Based on the suitability index, the model inference parameters are dynamically adjusted, a reinforcement learning adaptive parameter adjustment mechanism is introduced in the model inference process, a reward function is calculated according to the suitability evaluation result of the last period, and the model temperature parameter and weight coefficient are dynamically adjusted through the policy gradient algorithm; When the index is stable for multiple consecutive periods, the system automatically reduces the sampling frequency and the regulation amplitude.

[0016] Optionally, the updating regulation strategy specifically includes: Periodically collect teacher feedback data and regulation effectiveness data; The teacher feedback data and regulation effectiveness data are desensitized by a differential privacy mechanism; Based on the federal learning architecture, the LoRA parameters of each classroom node are aggregated to update the large language model; The confirmation data or correction data of the teacher on the regulation instruction are used as fine-tuning samples for the training process of the large model, forming a human-machine co-regulation closed-loop optimization mechanism; The optimization results are fed back to the data fusion and instruction generation link to form a closed loop and continuous learning mechanism.

[0017] In a second aspect, the present application provides a smart classroom suitability regulation system based on multi-modal and large language model, comprising: A data acquisition unit is configured to acquire multi-source data of the smart classroom classroom environment, wherein the multi-source data includes environmental modal data and behavioral modal data; A feature processing unit is configured to perform data preprocessing on the environmental modal data and behavioral modal data, extract key features of each modal, and construct a feature library of environmental features and behavioral features; A feature fusion unit is configured to fuse the environmental features and behavioral features using an orthogonal sequential fusion strategy to construct a fused feature vector; An intelligent analysis unit is configured to input the fused feature vector into a large language model to obtain a discrimination result of the classroom state, and generate an executable regulation instruction according to the discrimination result; the large language model is lightened by using low-rank adaptation technology, and the model size is reduced while maintaining the accuracy level through a quantization acceleration mechanism; A regulation feedback unit is configured to deliver the regulation instruction to the smart classroom central control system to realize real-time regulation of the environment and interactive reminders; A dynamic optimization unit is configured to collect environmental feedback data and student behavior feedback data after regulation, and update the regulation strategy based on the feedback data through a reinforcement learning algorithm to realize dynamic closed-loop optimization of suitability identification and regulation.

[0018] Optionally, the system adopts a cloud-edge collaborative architecture, including an edge node and a cloud node; The edge node is configured to execute the data acquisition unit; The cloud node is used for the execution of the feature processing unit, the feature fusion unit, the intelligent analysis unit, and the dynamic optimization unit.

[0019] Thirdly, this application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect or any possible implementation thereof.

[0020] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0021] Fifthly, this application provides a computer program product that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0022] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0023] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art: (1) This application uses multimodal data fusion technology to simultaneously collect environmental data and student behavior data, and constructs a multimodal classroom state perception system to achieve a comprehensive and accurate assessment of student suitability. Using preprocessing methods such as orthogonal fusion and feature alignment, heterogeneous data is transformed into structured feature vectors that can be efficiently processed by the model, simplifying the complex multi-source information processing process in traditional methods. Furthermore, this application can generate and execute control instructions in real time based on real-time analysis results and feed them back to the central control system for rapid control, thereby achieving real-time suitability control in the smart classroom scenario.

[0024] (2) The large language model of this application significantly reduces the computational burden and response latency of the model through low-rank adaptation and quantization acceleration technology. Furthermore, combined with the large language model fine-tuned by LoRA, it can generate personalized and operable control suggestions while identifying students' classroom status, thus realizing a closed loop from status analysis to intervention feedback.

[0025] (3) This application supports model quantization and acceleration, enhances the real-time application capability in smart classroom scenarios, and can reduce the burden of teachers’ post-class analysis, provide a visual and intelligent classroom suitability analysis tool, which helps to improve teaching quality and learning outcomes. Attached Figure Description

[0026] Figure 1 This is one of the flowcharts illustrating the smart classroom appropriateness control method based on multimodal and large language models provided in the embodiments of this application; Figure 2 This is the second flowchart of the smart classroom appropriateness control method based on multimodal and large language model provided in the embodiments of this application; Figure 3 This is the third flowchart of the smart classroom appropriateness control method based on multimodal and large language model provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the smart classroom appropriateness control device based on multimodal and large language model provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0028] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0029] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0030] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0031] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0032] The embodiments of this application are described below with reference to the accompanying drawings.

[0033] Reference Figure 1 This application provides a method for appropriate adjustment of smart classrooms based on multimodal and large language models, including: S101. Collect multi-source data of the smart classroom environment, including environmental modal data and behavioral modal data; S102. Perform data preprocessing on the environmental modal data and behavioral modal data, extract key features of each modality, and construct a feature library of environmental features and behavioral features; S103. The environmental features and behavioral features are fused using an orthogonal sequential fusion strategy to construct a fused feature vector; S104. Input the fused feature vector into the large language model to obtain the discrimination result of the classroom status, and generate executable control instructions based on the discrimination result; S105. Transmit the control commands to the smart classroom central control system to achieve real-time control of the environment and interactive reminders; S106. Collect environmental feedback data and student behavior feedback data after regulation, and update the regulation strategy based on the feedback data through reinforcement learning algorithm to achieve dynamic closed-loop optimization of appropriateness identification and regulation.

[0034] Specifically, in the data extraction step S101 of this embodiment, classroom environment data and student behavior data are extracted through a multi-source sensing module. Classroom environment data is collected through various sensors deployed in the classroom, such as temperature sensors, humidity sensors, light sensors, noise sensors, and air quality sensors, covering environmental parameters such as temperature, humidity, light intensity, noise levels, and carbon dioxide concentration. Student behavior data is collected through devices such as smart cameras and smart desk and chair sensors, including students' attention states (e.g., gaze direction, blinking frequency), participation levels (e.g., number of times hands are raised, frequency of speaking), and posture (e.g., whether sitting upright).

[0035] Specifically, this step enables real-time, synchronous perception of the classroom environment and student behavior, and includes: Hardware components: Environmental sensors: temperature, humidity, light intensity, CO2 concentration, noise / sound pressure level, PM2.5, etc.; Video acquisition unit: indoor fixed camera, fisheye / wide-angle optional; Microphone array for voice activity detection; Edge computing nodes for local preprocessing and encryption; Time synchronization unit supports NTP / PTP.

[0036] Sampling strategy and timing: Environmental sensors sample at a period of 0.2–1 Hz (example), and cameras capture raw video at 15–30 FPS; all data are timestamped (ISO 8601, millisecond level) at the acquisition end and written to a local circular buffer to support subsequent alignment.

[0037] Privacy and Preprocessing: Videos are anonymized / feature extracted at edge nodes. For example, human skeletal keypoints, gaze vectors, and expression category probabilities are uploaded only as abstract features (JSON / Protobuf) to avoid uploading raw face images; for audio, only volume, speech activity detection (VAD), and frequency domain energy features are saved, and the speech content is not saved.

[0038] Data transmission and protocols: MQTT / WebSocket / HTTP(S) are used from the edge to the cloud or management end; video streams can be H.264 encoded first, and keyframes are preserved and features extracted at the edge. JSON example for each message. Reliability: Multi-threaded acquisition and local buffering ensure no data loss during brief network interruptions; device heartbeat and health checks, such as heartbeat intervals and reconnection strategies, are used for operation and maintenance and robustness.

[0039] Step S102 involves preprocessing the collected multi-source data, including but not limited to the following steps: An outlier detection algorithm based on a sliding window is used to remove sensor drift and short-term noise from the multi-source data; a time-series alignment algorithm based on dynamic time warping is used to achieve time synchronization of the multi-source data; and a Z-Score normalization strategy is used to normalize the multi-source data to unify the feature scale.

[0040] In detail, this step transforms the output of heterogeneous sensors into high-quality time-series features that can be directly input into the model, specifically including: Outlier detection and denoising: Use sliding window + median filtering or Hampel filtering to remove burst values ​​from environmental sensors; use low-pass filtering (Butterworth) or Savitzky-Golay filtering to smooth continuous signals; use velocity / acceleration thresholding to detect abrupt changes and interpolate to repair them in visual keypoint sequences.

[0041] Missing value imputation: Short missing values ​​(< window length) are imputed using linear / spline interpolation; long missing values ​​are imputed using model prediction (simple regression or Kalman filtering based on similar nodes or historical patterns).

[0042] Unit and scale standardization: Standardize the units for all numerical features, such as light intensity (lx) and CO2 (ppm); then use Z-score (mean 0, variance 1) or Min-Max to normalize the features, and retain the normalization parameters for subsequent inverse normalization.

[0043] Time alignment, using a two-stage alignment: Phase 1: Hardware time synchronization (NTP / PTP) ensures consistency of the base timestamp; The second stage involves software-based sequence alignment. If there are differences in sampling rates or jitter, resampling or Dynamic Time Warping (DTW) is used to locally align the event sequences. The final output is a set of feature vectors with a fixed time window, such as a 5s / 10s / 30s window, and a configurable sliding step.

[0044] Feature engineering: Calculate statistical features such as mean, variance, maximum / minimum, slope, frequency domain features such as short-time Fourier transform energy spectrum, and behavioral features such as the number of times spoken per unit time, gaze distribution, and rate of change of posture.

[0045] Validation and formatting: A structured schema (such as JSON Schema / Protobuf) is used for field validation. Abnormal records are written to the log and trigger resampling / re-collection strategies.

[0046] In feature fusion step S103, classroom environment features and student behavior features are extracted, and the student behavior features and corresponding environmental features are fused using the Orthogonal Sequential Fusion (OSF) feature fusion method. First, features are extracted from the extracted classroom environment data and student behavior data to obtain their respective feature vectors. Then, orthogonalization techniques are used to map the environmental features and behavior features to unrelated subspaces, reducing feature redundancy. Next, environmental and behavior features are fused sequentially based on temporal priority. Finally, an attention weighting mechanism is used to optimize the fusion result, resulting in a fused feature vector.

[0047] Furthermore, utilizing model fine-tuning and input step S104, an evaluation system for the suitability of smart classrooms is established. Low-rank adaptation (LoRA) fine-tuning technology is employed to fine-tune the large language model in the field of smart education. The fusion features from step S103 are then input into the fine-tuned large language model. First, evaluation indicators for smart classroom suitability are determined, including dimensions such as comfort (based on environmental parameters), focus (based on behavioral parameters such as student attention), and interactivity (based on behavioral parameters such as student participation). Then, relevant data in the field of smart education is collected, and LoRA technology is used to perform lightweight fine-tuning of the large language model (such as the Llama series model) to adapt it to smart education scenarios. Finally, the fusion features obtained in step S102 are input into the fine-tuned large language model.

[0048] In this step, the integrated features are input into the large language model that has been fine-tuned using low-rank adaptation (LoRA) technology. This model has been specifically trained in the field of smart education and can analyze the matching degree between the current smart classroom and the students' status in real time based on the integrated features input and in combination with a preset classroom suitability evaluation system.

[0049] Based on real-time analysis results, the fine-tuned large language model generates corresponding control suggestions, such as adjusting classroom lighting, adjusting air conditioning temperature, and prompting teachers to increase interaction with students. These control suggestions are fed back to relevant modules of the smart classroom's central control system, as well as to the physical and teaching environments of the smart classroom, to guide subsequent environmental adjustments and teaching activities, thus achieving dynamic optimization of the smart classroom.

[0050] After fine-tuning the large language model based on the integrated feature inputs, the system enters the classroom suitability evaluation and control phase. The model calculates a comprehensive classroom suitability index based on student behavioral characteristics such as attention, participation, and posture, and environmental characteristics such as lighting, temperature, humidity, noise, and air quality. This index covers dimensions such as comfort, focus, and interaction, reflecting the degree to which the current teaching environment supports the learning state. Subsequently, the model generates control instructions based on the suitability level, such as adjusting lighting, ventilation, temperature, humidity, or the pace of teacher interaction.

[0051] Furthermore, the regulatory instructions include at least one of the following: Environmental control commands are used to regulate lighting, air quality, temperature, humidity, and ventilation systems. Interactive prompts are used to remind teachers or students to adjust their posture, focus their attention, or promote teacher-student interaction. Adaptive feedback instructions are used to guide the system to adjust the sampling frequency or control amplitude to optimize energy consumption and response latency.

[0052] In step S105, the control commands are transmitted to the smart classroom central control system via the closed-loop feedback module for execution. Simultaneously, the execution results and student feedback are recorded, enabling dynamic strategy optimization. This step ensures that the smart classroom can optimize teaching conditions in real time and adaptively, improving student focus and learning efficiency.

[0053] Finally, in step S106, feedback results are collected and the control strategy is automatically updated to achieve dynamic closed-loop optimization of appropriateness identification and control. The system collects feedback results and automatically updates the control strategy to achieve dynamic closed-loop optimization of appropriateness identification and control.

[0054] This step describes how to translate feedback into updates to the model and policy, specifically: Data collection and indicator calculation: After the loop is closed, environmental and behavioral change data are collected, and a set of evaluation indicators are calculated, such as the rate of change in attention, the amount of change in participation, the score of emotional stability, and the change in energy consumption. These are used as "rewards / labels" for subsequent learning.

[0055] Online / Offline Learning Process: Online: Lightweight strategy gradient or Bandit algorithm is used to make small real-time adjustments to decision parameters, such as adjusting the LLM output temperature or controlling priority weights, to ensure that the system adapts to short-term environmental changes; Offline: After periodically collecting data, LoRA is fine-tuned or the policy network is trained, and updates are distributed in the cloud or on a secure training cluster, using A / B testing or Canary release methods.

[0056] Privacy and Compliance: Logs reported for training must undergo differential privacy / anonymization processing. If federated learning is used, only model gradients or LoRA parameter updates should be uploaded to protect the original data.

[0057] Human-machine collaborative learning: Teachers can confirm / correct the system's suggestions, and these human-generated labels are saved and used to supervise training to improve interpretability and acceptability.

[0058] Reliability and Rollback: All model updates have rollback mechanisms and version management; offline simulation evaluation and small-scale tests are conducted before updates to confirm that there are no abnormal indicators and rollback is before full deployment.

[0059] Evaluation cycle and monitoring: Set KPIs (identification accuracy, response latency, average energy consumption) and display them in real time on the monitoring panel. If the KPI drops, an alarm will be automatically triggered and the automatic update process will be paused.

[0060] Optionally, the collection of multi-source data on the smart classroom environment includes: Temperature, humidity, light intensity, carbon dioxide concentration, and noise level data are collected using an environmental data acquisition unit and used as the environmental modal data. Using a behavioral vision acquisition unit, image recognition algorithms are used to capture students’ posture, gaze direction, facial expressions and speaking frequency as behavioral modality data. A multi-threaded synchronization mechanism is used to ensure the time consistency between environmental modal data and behavioral modal data, thereby achieving multi-modal acquisition synchronization.

[0061] Specifically, the multi-source perception module includes an environmental data acquisition unit, a behavior vision acquisition unit, and a multi-threaded synchronization mechanism, wherein: The environmental data acquisition unit is used to collect environmental information such as temperature, humidity, light intensity, carbon dioxide concentration, and noise level; the behavioral vision acquisition unit is used to capture behavioral characteristics such as student posture, gaze direction, facial expression, and speaking frequency through image recognition algorithms; the multi-threaded synchronization mechanism is used to ensure the temporal consistency of environmental data and behavioral vision data, and to achieve multimodal acquisition synchronization.

[0062] Optionally, behavioral modality data extraction specifically includes: extracting key point data of students based on human posture estimation algorithms, calculating joint angles to distinguish classroom behaviors such as sitting posture, standing posture, and raising hands; analyzing students' emotional states, such as focus, drowsiness, and confusion, based on facial expression recognition models; and determining students' gaze direction based on eye movement detection to judge whether they are paying attention to the teacher or teaching content.

[0063] Environmental modal data extraction includes: using a gas sensor to obtain the carbon dioxide concentration in the classroom, and prompting ventilation when the concentration exceeds a preset threshold; using a light sensor to monitor the ambient brightness in real time, and judging whether the lighting is appropriate based on the clarity of students' facial features; and using a noise sensor to detect the background noise level, and generating control suggestions to reduce noise interference when the noise exceeds a set threshold.

[0064] Optionally, the process of obtaining the fused feature vector includes: Map environmental and behavioral characteristics to unrelated subspaces; Based on temporal priority, environmental features and behavioral features are fused sequentially to obtain preliminary fusion results; The fusion result is optimized by an attention weighting mechanism to obtain the final fusion feature vector.

[0065] In this embodiment, environmental data results and behavioral data results are input into the orthogonal serialization feature fusion module. First, the environmental features and behavioral features are orthogonalized so that the two types of features are mapped to unrelated subspaces to reduce redundancy. Then, the environmental features and behavioral features are fused in sequence according to temporal priority. The fusion process is optimized through an attention weighting mechanism to obtain the fused comprehensive features.

[0066] For example, the data fusion in this application is achieved through an orthogonal sequential fusion module, which includes a feature orthogonalization unit, a feature sequential fusion unit, and a fusion output unit. The feature orthogonalization unit maps environmental features and behavioral features to unrelated subspaces to reduce redundancy. The feature sequential fusion unit fuses environmental and behavioral features sequentially based on temporal priority. The fusion output unit optimizes the fusion result through an attention weighting mechanism and outputs the fused feature vector to a large language model.

[0067] It should be noted that, in this embodiment, before the fusion feature is input into the large language model, model acceleration methods such as quantization, distillation, or pruning can be used to improve inference efficiency in resource-constrained smart classroom devices.

[0068] Optionally, the large language model is built on Llama 8B; the large language model is lightweighted and fine-tuned by using low-rank adaptation technology, and the model size is reduced while maintaining the accuracy level by using a quantization acceleration mechanism.

[0069] Specifically, the large language model fine-tuning module is based on Llama 8B and uses low-rank adaptation (LoRA) technology for lightweight fine-tuning. It also uses a quantization acceleration mechanism to reduce the model accuracy from FP16 to INT4 to reduce computational latency and memory usage. At the same time, it combines human feedback reinforcement learning (RLHF) strategy to optimize the generation results of control instructions.

[0070] Optionally, the method further includes: Calculate the appropriateness indicators for students in the classroom, including attention concentration, emotional stability, participation, and environmental comfort; The model inference parameters are dynamically adjusted based on the suitability index. A reinforcement learning adaptive parameter tuning mechanism is introduced during the model inference process. The reward function is calculated based on the suitability evaluation results of the previous cycle, and the model temperature parameters and weight coefficients are dynamically adjusted through the policy gradient algorithm. When the indicators stabilize over multiple consecutive periods, the system automatically reduces the sampling frequency and adjustment amplitude.

[0071] Specifically, the indicator calculation process in this application embodiment is implemented through a suitability assessment module. The suitability assessment module calculates student classroom suitability indicators, including attention concentration, emotional stability, participation, and environmental comfort. Based on these indicators, it dynamically adjusts the model inference parameters, making system control more personalized and refined.

[0072] In the model inference process, a reinforcement learning adaptive parameter tuning mechanism is introduced. The reward function is calculated based on the suitability evaluation results of the previous cycle, and the model temperature parameters and weight coefficients are dynamically adjusted through the policy gradient algorithm. When the indicators are stable for several consecutive cycles, the system automatically reduces the sampling frequency and adjustment amplitude to achieve energy consumption optimization.

[0073] Optionally, the update control strategy specifically includes: Periodically collect teacher feedback data and data on the effectiveness of regulation; Differential privacy mechanisms are used to anonymize teacher feedback data and data on the effectiveness of regulation. Based on the federated learning architecture, the LoRA parameters of each classroom node are aggregated to update the large language model; The teacher's confirmation or correction data of the control instructions are used as fine-tuning samples for the training process of the large model, forming a closed-loop optimization mechanism of human-machine co-tuning. The optimization results are fed back to the data fusion and instruction generation stages, forming a closed-loop and continuous learning mechanism.

[0074] Specifically, the embodiments of this application provide a self-learning and model update capability, including periodically collecting teacher feedback and adjustment effectiveness data, desensitizing it through a differential privacy mechanism and uploading it to the cloud, and aggregating LoRA parameters based on a federated learning architecture to achieve model updates; teachers can confirm or correct the system output instructions, and the corrected data will be used as fine-tuning samples input into the model to achieve a closed-loop optimization mechanism of human-machine co-adjustment, wherein the optimization results are fed back to the data fusion and decision-making process to form a dependent closed loop and continuous learning mechanism.

[0075] This application's embodiments construct a complete technical system, forming an adaptive closed-loop structure from multi-source perception, feature fusion, model fine-tuning to feedback regulation and relearning. This enables continuous evolution of the system in the perception, decision-making, and optimization stages. This application can achieve fine-grained identification and real-time adjustment of student classroom suitability in smart classrooms, significantly improving the intelligence level of the teaching environment and human-machine collaboration capabilities.

[0076] Reference Figure 2 The complete process of this application embodiment includes the following steps: Environmental and behavioral data collection: Data collection is carried out on the physical and teaching environment of the smart classroom. On the one hand, through the environmental data monitoring module, various sensors such as temperature sensors, humidity sensors, light sensors, and noise sensors are used to collect environmental data such as classroom temperature, humidity, light intensity, and noise decibels. On the other hand, through the student behavior recognition module, with the help of smart cameras, motion capture equipment, etc., behavioral data such as students' posture, gaze direction, facial expressions, and speaking frequency are identified.

[0077] Environmental data processing: The collected environmental data is fused to integrate data from multiple environmental sensors, remove noise, fill in missing values, unify the format of environmental data from different dimensions and correlate them to obtain environmental data results. These results can comprehensively reflect the overall condition of the classroom's physical environment.

[0078] Behavioral data processing: Conduct behavioral data analysis on student behavioral data, using computer vision algorithms, machine learning models, etc., to extract effective features from raw behavioral data, analyze students' attention concentration, participation, emotional state, etc., and obtain behavioral data results to characterize students' behavior in the classroom.

[0079] Feature fusion: The environmental data results and behavioral data results are input into the orthogonal serialization feature fusion module. First, the environmental features and behavioral features are orthogonalized so that the two types of features are mapped to unrelated subspaces to reduce redundancy. Then, the environmental features and behavioral features are fused in sequence according to temporal priority. The fusion process is optimized through an attention weighting mechanism to obtain the fused comprehensive features.

[0080] Real-time model analysis: The integrated features are input into a large language model that has been fine-tuned using low-rank adaptation (LoRA) technology. This model has been specifically trained in the field of smart education and can analyze the degree of matching between the current smart classroom and the students' status in real time based on the integrated features input and a pre-set classroom suitability evaluation system.

[0081] Regulation suggestion generation and feedback: Based on real-time analysis results, the fine-tuned large language model generates corresponding regulation suggestions, such as adjusting classroom lighting, adjusting air conditioning temperature, and prompting teachers to increase interaction with students. These suggestions are fed back to relevant modules of the smart classroom's central control system, as well as to the physical and teaching environments of the smart classroom, to guide subsequent environmental adjustments and teaching activities, achieving dynamic optimization of the smart classroom.

[0082] Reference Figure 3 , Figure 3 This is a flowchart of an embodiment of this application combined with a specific system, including the following specific steps: Step 1: Conduct environmental data collection. Utilize various sensors deployed in the smart classroom, such as temperature sensors, humidity sensors, and light sensors, to collect real-time physical environmental parameters such as temperature, humidity, and light intensity, providing foundational environmental data for subsequent analysis. This step is an independent initial step and does not depend on other preliminary steps.

[0083] Step Two: Simultaneously collect student behavior data. Using smart cameras, wearable devices, and other means, collect data on students' posture, concentration, and other classroom behaviors, creating a complementary multi-source dataset with environmental data. This step is a parallel initial step with Step One; they are not directly dependent on each other and together form the foundation for data collection.

[0084] Step 3: Constructing the fine-tuning dataset. The collected environmental and student behavior data undergo preprocessing, including outlier removal and missing value imputation, before being organized into a dataset suitable for fine-tuning the large language model according to specific formats and annotation requirements. This step is based on the environmental data collected in Step 1 and the student behavior data collected in Step 2, and involves processing the raw data from the first two steps.

[0085] Step 4: Implement memory access optimization for CUDA operators. This step optimizes the way model data is stored and retrieved in memory, reducing data transfer latency, improving data access efficiency, and providing memory-level support for efficient model operation. This step is a separate preparatory step for model optimization and does not directly depend on the first three steps, but it serves subsequent model processing stages.

[0086] Step 5: Optimize the computational pipeline of CUDA operators. Decompose the model computation task into multiple parallelizable pipeline stages to increase computational throughput and accelerate data processing. This step builds upon the memory access optimization in Step 4, further enhancing the computational efficiency of CUDA operators.

[0087] Step 6: Perform whole-network optimization of the CUDA operator model. Adjust the overall structure of the large language model, remove redundant computational parts, enhance the overall performance of the model, and further improve data processing efficiency. This step is carried out based on the computational pipeline optimization in Step 5, achieving overall optimization of the model structure.

[0088] Step 7: Perform OSF-based unimodal performance ranking. Evaluate and rank the performance of the environmental and student behavior unimodal data to clarify their respective weights in subsequent fusion. This step analyzes the basic performance of the two unimodal data based on the environmental data from Step 1 and the student behavior data from Step 2.

[0089] Step 8: Implement OSF-based sequential fusion design. Determine the order in which environmental features and student behavioral features are fused to make feature fusion more logical and effective. This step is based on the single-modal performance ranking results from Step 7, and determines the fusion order according to the ranking.

[0090] Step 9: Introduce an OSF-based orthogonal loss function. This makes the fused environmental and behavioral features more discriminative and effective, improving the quality of feature fusion. Building upon the sequential fusion design determined in Step 8, this step optimizes the fusion effect through the loss function.

[0091] Step 10: Insert a low-rank adaptation module into the large language model and conduct downstream task adaptation training. This lightweight approach allows the model to learn specific task knowledge while adapting to the downstream task of smart classroom regulation, learning the correlation between the environment and student behavior. This step integrates the fine-tuned dataset built in Step 3, the model's overall network optimization results from Step 6, and the feature fusion optimization results from Step 9, training based on the combined data and model optimization foundation.

[0092] Step 11: Input the processed environmental and behavioral data into the fine-tuned large language model. Based on the learned patterns, the model generates control suggestions for the current smart classroom, such as adjusting environmental parameters and providing teaching interaction prompts. This step builds upon the model training completed in Step 10, using the optimized model to process the newly input data.

[0093] Step 12: Feedback and Closed Loop Formation of Control Results. After the control recommendations are implemented, environmental change and student behavior feedback data are collected and fed back to the data collection stage. This data then enters subsequent processes such as dataset construction and model optimization to continuously improve the accuracy and effectiveness of control. This step, based on the control recommendations and implementation results generated in Step 11, feeds the feedback data back to the environmental data collection in Step 1 and the student behavior data collection in Step 2, forming a complete closed loop.

[0094] It should be further noted that the CUDA operators in this embodiment are specifically optimized core computation functions executed on NVIDIA GPUs. During the inference process of the large language model, each computation process is defined as a specific CUDA operator, utilizing GPU parallel computation to meet real-time requirements.

[0095] Reference Figure 4 This application provides a smart classroom appropriateness control system based on multimodal and large language models, comprising: The data acquisition unit 410 is used to collect multi-source data of the smart classroom environment, including environmental modal data and behavioral modal data. Feature processing unit 420 is used to preprocess the environmental modal data and behavioral modal data, extract key features of each modality, and construct a feature library of environmental features and behavioral features; The feature fusion unit 430 is used to fuse the environmental features and behavioral features using an orthogonal sequential fusion strategy to construct a fused feature vector. The intelligent analysis unit 440 is used to input the fused feature vector into the large language model to obtain the discrimination result of the classroom state, and generate executable control instructions based on the discrimination result; the large language model is lightweightly fine-tuned by using low-rank adaptation technology, and the model size is reduced while maintaining the accuracy level through a quantization acceleration mechanism. The control feedback unit 450 is used to transmit control commands to the smart classroom central control system to realize real-time control of the environment and interactive reminders; The dynamic optimization unit 460 is used to collect environmental feedback data and student behavior feedback data after regulation, and update the regulation strategy based on the feedback data through reinforcement learning algorithm to realize dynamic closed-loop optimization of appropriateness identification and regulation.

[0096] In an optional example, the intelligent analysis unit can be further integrated with a classroom interaction behavior database to achieve adaptive regulation of student status under different teaching modes, such as outputting differentiated regulation schemes in group discussions, experimental operations, or teacher lecture modes.

[0097] The control feedback unit can directly execute control measures through the intelligent classroom's environmental control system, such as automatically adjusting lights, turning the air conditioner on or off, and sending prompts to the teacher's terminal.

[0098] Optionally, the system adopts a cloud-edge collaborative architecture, including edge nodes and cloud nodes; The edge node is used for the execution of the data acquisition unit; The cloud node is used for the execution of the feature processing unit, the feature fusion unit, the intelligent analysis unit, and the dynamic optimization unit.

[0099] Reference Figure 5 Based on the methods in the above embodiments, this application provides an electronic device that may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540. The processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions stored in the memory 530 to execute the methods in the above embodiments.

[0100] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0101] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0102] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0103] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0104] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0105] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0106] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0107] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A multi-modal and large language model-based wisdom classroom appropriateness regulation method, characterized in that, The method comprises the following steps: Collecting multi-source data of the classroom environment in the smart classroom, including environmental modal data and behavioral modal data; Data preprocessing of the environmental modal data and behavioral modal data, extracting key features of each modal, and constructing a feature library of environmental features and behavioral features; Fusing the environmental features and behavioral features using an orthogonal sequential fusion strategy to construct a fusion feature vector; Inputting the fusion feature vector into a large language model to obtain a discrimination result of the classroom state, and generating executable control instructions according to the discrimination result; Delivering the control instructions to the central control system of the smart classroom to realize real-time control of the environment and interactive reminders; Collecting feedback data of the environment and student behavior after control, and updating the control strategy based on the feedback data through a reinforcement learning algorithm to realize dynamic closed-loop optimization of suitability identification and control.

2. The multi-modal and large language model-based smart classroom suitability regulation method according to claim 1, characterized in that, The collection of multi-source data of the classroom environment in the smart classroom comprises: Using an environmental data collection unit to collect temperature, humidity, light intensity, carbon dioxide concentration and noise level data as the environmental modal data; Using a behavior visual collection unit, capturing student posture, gaze direction, facial expression and speaking frequency through image recognition algorithm as behavioral modal data; Using a multi-thread synchronization mechanism to ensure the time consistency of the environmental modal data and the behavioral modal data, and realizing multi-modal collection synchronization.

3. The multi-modal and large language model-based smart classroom suitability regulation method according to claim 1, characterized in that, The data preprocessing process comprises: Based on the sliding window outlier detection algorithm, the sensor drift and short-term noise in the multi-source data are removed; Based on the time series alignment algorithm of dynamic time warping, the time synchronization of the multi-source data is realized; Based on the Z-Score standardization strategy, the multi-source data is normalized to unify the feature scale.

4. The multi-modal and large language model-based smart classroom suitability regulation method according to claim 1, characterized in that, The acquisition process of the fusion feature vector comprises: Mapping the environmental features and behavioral features to subspaces that are not related to each other; Based on the time sequence priority, the environmental features and behavioral features are fused in turn to obtain a preliminary fusion result; Optimizing the fusion result through attention weighting mechanism to obtain the final fusion feature vector.

5. The multi-modal and large language model-based smart classroom suitability regulation method according to claim 1, wherein, The control instructions include at least one of the following: Environment control instructions for adjusting light, air quality, temperature and humidity, and ventilation system; Interactive prompt instructions for reminding teachers or students to adjust posture, concentrate attention or promote teacher-student interaction; Adaptive feedback instructions for guiding the system to adjust the sampling frequency or control amplitude to optimize energy consumption and response time delay.

6. The multi-modal and large language model-based smart classroom suitability regulation method according to claim 1, characterized in that, The large language model is constructed based on Llama 8B; the large language model is lightened by using low-rank adaptation technology, and the model size is reduced while maintaining the accuracy level through quantization acceleration mechanism.

7. The multi-modal and large language model-based smart classroom suitability regulation method according to claim 1, characterized in that, The method further comprises: Calculating the suitability index of students in the classroom, including attention concentration, emotional stability, participation and environmental comfort; Based on the suitability index, dynamically adjusting the model inference parameters, introducing a reinforcement learning adaptive parameter adjustment mechanism in the model inference process, calculating the reward function according to the suitability evaluation result of the last period, and dynamically adjusting the model temperature parameter and weight coefficient through the policy gradient algorithm; When the continuous multi-period indicators are stable, the system automatically reduces the sampling frequency and the regulation amplitude.

8. The multi-modal and large language model-based smart classroom suitability regulation method according to claim 1, characterized in that, The updating regulation strategy specifically includes: Periodically collecting teacher feedback data and regulation effectiveness data; Through the differential privacy mechanism, the teacher feedback data and the regulation effectiveness data are desensitized; Based on the federated learning architecture, the LoRA parameters of each classroom node are aggregated to update the large language model; The teacher's confirmation data or correction data of the regulation instruction is used as a fine-tuning sample for the training process of the large model, forming a human-machine co-regulation closed-loop optimization mechanism; The optimization results are fed back to the data fusion and instruction generation link to form a closed loop and continuous learning mechanism.

9. A multi-modal and large language model-based smart classroom suitability regulation system, characterized in that, It includes: A data acquisition unit for acquiring multi-source data of the classroom environment of the smart classroom, the multi-source data including environmental modal data and behavioral modal data; A feature processing unit for pre-processing the environmental modal data and behavioral modal data, extracting key features of each modal, and constructing a feature library of environmental features and behavioral features; A feature fusion unit for fusing the environmental features and behavioral features using an orthogonal sequential fusion strategy to construct a fused feature vector; An intelligent analysis unit for inputting the fused feature vector into a large language model to obtain a discrimination result of the classroom state, and generating an executable regulation instruction based on the discrimination result; The large language model is fine-tuned by using low-rank adaptation technology, and the model size is reduced while maintaining the accuracy level through a quantization acceleration mechanism; A regulation feedback unit for delivering the regulation instruction to the smart classroom central control system to realize real-time regulation of the environment and interactive reminders; A dynamic optimization unit for collecting environmental feedback data and student behavior feedback data after regulation, and updating the regulation strategy based on the feedback data through a reinforcement learning algorithm to realize dynamic closed-loop optimization of adaptability identification and regulation.

10. The smart classroom adaptability regulation system based on multi-modal and large language model according to claim 9, the system adopts a cloud-edge collaborative architecture, including an edge node and a cloud node; The edge node is used for the execution of the data acquisition unit; The cloud node is used for the execution of the feature processing unit, the execution of the feature fusion unit, and the execution of the intelligent analysis unit and the dynamic optimization unit.

Images