Classroom environment intelligent monitoring method and system based on data analysis

Abstract

The application discloses a classroom environment intelligent monitoring method and system based on data analysis, and the method comprises the following steps: acquiring multi-source heterogeneous data in a classroom environment, eliminating sensor noise of the multi-source heterogeneous data by using an improved Kalman filtering algorithm, performing space-time alignment on the data after noise elimination, and obtaining aligned multi-source heterogeneous data; establishing a CRNN three-layer convolutional recurrent neural network model, inputting the aligned multi-source heterogeneous data into the CRNN three-layer convolutional recurrent neural network model for identification, and obtaining a classroom teaching environment state; and performing classroom risk early warning and self-adaptive classroom environment regulation and control according to the classroom teaching environment state. The method realizes active prevention of classroom sudden risks, can automatically adjust an intelligent air conditioner and a lighting system, creates a safe and comfortable teaching environment, and is beneficial to classroom efficiency improvement and teacher and student health guarantee.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of big data analysis, in particular to a classroom environment intelligent monitoring method and system based on data analysis. BACKGROUND

[0002] With the development of big data analysis technology, data analysis is also widely used in classroom monitoring. However, the current classroom monitoring traditional video monitoring only records the picture, and lacks the ability of behavior semantic analysis. And the environmental control relies on fixed threshold, ignoring the dynamic demand of teaching scene, and teachers cannot obtain the quantitative feedback of the overall learning state of the class in real time. Although the existing technology can detect the posture of students, it has high misjudgment rate due to single mode data analysis, and lacks correlation analysis of teaching process and linearization of control strategy in data analysis. Therefore, how to improve the efficiency and accuracy in traditional teaching monitoring is a technical problem to be solved at present. SUMMARY

[0003] The purpose of the present application is to solve the above problems, and a classroom environment intelligent monitoring method and system based on data analysis are designed.

[0004] To achieve the above purpose, the technical scheme of the present application is as follows: further, in the above-mentioned classroom environment intelligent monitoring method based on data analysis, the classroom environment intelligent monitoring method comprises the following steps:

[0005] Obtain multi-source heterogeneous data in the classroom environment, wherein the multi-source heterogeneous data at least includes environment image data, environment sound data and environment temperature and humidity data;

[0006] Eliminate the sensor noise of the multi-source heterogeneous data by using the improved Kalman filtering algorithm, and perform time and space alignment on the data after noise elimination to obtain aligned multi-source heterogeneous data;

[0007] Establish a CRNN three-layer convolutional recurrent neural network model, input the aligned multi-source heterogeneous data into the CRNN three-layer convolutional recurrent neural network model for recognition, and obtain the classroom teaching environment state;

[0008] According to the classroom teaching environment state, the classroom risk early warning and self-adaptive classroom environment control are carried out.

[0009] Further, in the above-mentioned classroom environment intelligent monitoring method based on data analysis, the multi-source heterogeneous data in the classroom environment is obtained, and the multi-source heterogeneous data at least includes environment image data, environment sound data and environment temperature and humidity data, which comprises:

[0010] Obtain environment image data in the classroom environment through an image sensor, wherein the environment image data at least includes color image, depth information and backlight scene;

[0011] A channel ring microphone array was used to acquire individual student speech data, teacher lecturing voice, student discussion voice, environmental noise, and writing friction sound in the classroom environment to obtain environmental sound data.

[0012] Temperature and humidity data in the classroom environment are obtained by using temperature and humidity sensors.

[0013] Furthermore, in the aforementioned intelligent classroom environment monitoring method based on data analysis, the step of using an improved Kalman filtering algorithm to eliminate sensor noise in the multi-source heterogeneous data, and then performing spatiotemporal alignment on the noise-reduced data to obtain aligned multi-source heterogeneous data, includes:

[0014] An improved Kalman filtering algorithm is used to eliminate sensor noise in the multi-source heterogeneous data. An adaptive noise covariance matrix is ​​established for different sensor characteristics, and the filtering parameters are dynamically adjusted.

[0015] By aligning the timestamps of image, sound, and temperature / humidity data, and achieving cross-modal data association through extended state vectors, audio energy fluctuations are analyzed synchronously with student actions in videos to obtain aligned multi-source heterogeneous data.

[0016] Furthermore, in the aforementioned intelligent classroom environment monitoring method based on data analysis, the step of using an improved Kalman filtering algorithm to eliminate sensor noise in the multi-source heterogeneous data and performing spatiotemporal alignment on the noise-reduced data to obtain aligned multi-source heterogeneous data further includes:

[0017] Based on the PTP precise time protocol, devices in the noise-reduced multi-source heterogeneous data are always synchronized. The network latency difference is eliminated by hardware timestamps to obtain time-aligned data.

[0018] The physical locations of the sensor nodes in the time-aligned data are mapped to the camera coordinate system. The correspondence between the desk and the image pixels is established through the QR code calibration board to obtain environmental mapping data. The tracking path of the PTZ camera is dynamically corrected using the sound source localization results to obtain visual association data.

[0019] By tracking key points in the skeleton, the combined actions of standing and talking and dozing off at the table are identified, and the duration is determined by combining time series analysis; a comprehensive scoring model is constructed by analyzing facial orientation, gaze focus and micro-expressions.

[0020] Furthermore, in the aforementioned intelligent classroom environment monitoring method based on data analysis, the establishment of the CRNN three-layer convolutional recurrent neural network model includes:

[0021] The first layer of the CRNN three-layer convolutional recurrent neural network model is used to extract image edge features, the second layer is used to identify dynamic behavior, and the third layer is used to fuse temporal data.

[0022] A bidirectional LSTM network is introduced into the recurrent layer of the model to process the temporal dependencies of sound signals and capture the patterns of environmental state changes.

[0023] The CRNN three-layer convolutional recurrent neural network model was trained using a historical classroom environment dataset in the database. The importance of different modalities was weighted through an attention mechanism to obtain the trained CRNN three-layer convolutional recurrent neural network model.

[0024] Furthermore, in the aforementioned intelligent classroom environment monitoring method based on data analysis, the step of inputting the aligned multi-source heterogeneous data into the CRNN three-layer convolutional recurrent neural network model for identification to obtain the classroom teaching environment status includes:

[0025] The classroom teaching environment status includes at least the teaching behavior recognition status and the environmental anomaly detection status;

[0026] The quality of teacher-student interaction is judged based on the activity level of group discussions, and the student's focus is analyzed through facial orientation and eye focus to obtain the teaching behavior recognition status.

[0027] The system identifies abnormal environmental conditions by detecting equipment malfunctions, safety hazards, and emergencies in the classroom environment.

[0028] Furthermore, in the aforementioned intelligent classroom environment monitoring method based on data analysis, the step of conducting classroom risk warning and adaptive classroom environment control based on the state of the classroom teaching environment includes:

[0029] The system triggers audible and visual alarms in response to fires or sudden loud noise events, and pushes information to the administrator's terminal.

[0030] Based on the duration of temperature and humidity deviations from the comfort zone and the proportion of students distracted, a three-level warning system (yellow / orange / red) is generated to prompt teachers to adjust their teaching strategies.

[0031] The system uses a PID algorithm to control the air conditioning and lighting systems. When it detects a drop in student concentration, it pushes interactive courseware or switches the teaching scene.

[0032] Furthermore, in the data analysis-based intelligent classroom environment monitoring system, the intelligent classroom environment monitoring system includes the following modules:

[0033] The environmental data acquisition module is used to acquire multi-source heterogeneous data in the classroom environment. The multi-source heterogeneous data includes at least environmental image data, environmental sound data, and environmental temperature and humidity data.

[0034] The data alignment processing module is used to eliminate sensor noise in the multi-source heterogeneous data using an improved Kalman filtering algorithm, and to perform spatiotemporal alignment on the noise-eliminated data to obtain aligned multi-source heterogeneous data.

[0035] The environmental state recognition module is used to establish a CRNN three-layer convolutional recurrent neural network model, input the aligned multi-source heterogeneous data into the CRNN three-layer convolutional recurrent neural network model for recognition, and obtain the classroom teaching environment state;

[0036] The classroom environment monitoring module is used to provide early warnings of classroom risks and adaptive classroom environment control based on the status of the classroom teaching environment.

[0037] Furthermore, in the intelligent classroom environment monitoring system based on data analysis, the data encryption module includes the following sub-modules:

[0038] The image data acquisition submodule is used to acquire environmental image data in the classroom environment through an image sensor. The environmental image data includes at least color images, depth information, and backlighting scenes.

[0039] The sound data acquisition submodule is used to acquire individual student speech data, teacher lecturing voice, student discussion voice, environmental noise and writing friction sound in the classroom environment using a channel ring microphone array, to obtain environmental sound data.

[0040] The temperature and humidity data acquisition submodule is used to acquire temperature and humidity data in the classroom environment through temperature and humidity sensors, thereby obtaining environmental temperature and humidity data.

[0041] Furthermore, in the intelligent classroom environment monitoring system based on data analysis, the data processing module includes the following sub-modules:

[0042] The time alignment submodule is used to synchronize devices in noise-reduced multi-source heterogeneous data based on the PTP precise time protocol. It eliminates network latency differences through hardware timestamps to obtain time-aligned data.

[0043] The spatial alignment submodule is used to map the physical positions of sensor nodes in the time alignment data to the camera coordinate system, establish the correspondence between the desk and image pixels through the QR code calibration board to obtain environmental mapping data, and dynamically correct the tracking path of the PTZ camera using the sound source localization results to obtain visual association data.

[0044] The data processing and analysis submodule is used to identify the combined actions of standing and talking and dozing off by tracking the skeleton key points, and to determine the duration by combining time series analysis. It also analyzes facial orientation, gaze focus and micro-expressions to build a comprehensive scoring model.

[0045] Its beneficial effects lie in the construction of a three-dimensional monitoring system through a multi-source heterogeneous data acquisition mechanism. The solution integrates multi-modal data such as environmental images, sound, temperature, and humidity, overcoming the limitations of single-sensor monitoring and comprehensively capturing classroom dynamics. It can capture students' attention states through image recognition, identify abnormal sounds through sound analysis, and assess environmental comfort in real time by combining temperature and humidity data, providing comprehensive data support for classroom monitoring. Secondly, the improved Kalman filtering algorithm and spatiotemporal alignment technology ensure data quality. Addressing sensor noise issues, the filtering model is optimized to improve the data signal-to-noise ratio and avoid misjudgments; spatiotemporal alignment processing eliminates time differences and spatial deviations in multi-source data, ensuring that images, sound, and environmental parameters are fused in a unified spatiotemporal coordinate system, laying a reliable foundation for subsequent intelligent analysis. Finally, the CRNN model and adaptive control form a closed-loop management system. Utilizing the spatiotemporal feature extraction capabilities of convolutional recurrent neural networks, the system accurately identifies classroom environmental states and triggers risk warnings and environmental controls in real time. This mechanism not only proactively prevents unexpected risks in the classroom, but also creates a safe and comfortable teaching environment through automatic adjustment of intelligent air conditioning and lighting systems, which is conducive to improving classroom efficiency and protecting the health of teachers and students. Attached Figure Description

[0046] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0047] Figure 1 This is a schematic diagram of the first embodiment of the intelligent classroom environment monitoring method based on data analysis in this invention.

[0048] Figure 2 This is a schematic diagram of the second embodiment of the intelligent classroom environment monitoring method based on data analysis in this invention.

[0049] Figure 3 This is a schematic diagram of the first embodiment of the intelligent classroom environment monitoring system based on data analysis in this invention.

[0050] Figure 4 This is a schematic diagram of the CRNN three-layer recursive convolutional neural network model of the intelligent classroom environment monitoring method based on data analysis in an embodiment of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0052] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0053] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1 As shown, a data-driven intelligent classroom environment monitoring method includes the following steps:

[0054] Step 101: Obtain multi-source heterogeneous data in the classroom environment, wherein the multi-source heterogeneous data includes at least environmental image data, environmental sound data, and environmental temperature and humidity data;

[0055] Specifically, in this embodiment, environmental image data in the classroom environment is acquired through an image sensor. The environmental image data includes at least color images, depth information, and backlighting.

[0056] A channel ring microphone array was used to acquire individual student speech data, teacher lecturing voice, student discussion voice, environmental noise, and writing friction sound in the classroom environment to obtain environmental sound data.

[0057] Temperature and humidity data in the classroom environment are obtained by using temperature and humidity sensors.

[0058] 1.1 The present invention will now be described in detail with reference to the accompanying drawings. Figure 1 As shown, a data-driven intelligent classroom environment monitoring method includes the following steps:

[0059] Step 101: Obtain multi-source heterogeneous data in the classroom environment, wherein the multi-source heterogeneous data includes at least environmental image data, environmental sound data, and environmental temperature and humidity data;

[0060] Specifically, in this embodiment, environmental image data in the classroom environment is acquired through an image sensor. The environmental image data includes at least color images, depth information, and backlighting.

[0061] A channel ring microphone array was used to acquire individual student speech data, teacher lecturing voice, student discussion voice, environmental noise, and writing friction sound in the classroom environment to obtain environmental sound data.

[0062] Temperature and humidity data in the classroom environment are obtained by using temperature and humidity sensors.

[0063] 1.2 Deployment of Distributed Acoustic Sensing Layer Equipment: Microphone Array: An 8-channel circular microphone array, arranged in a double-layer layout of "4 inside, 4 outside" (inner layer diameter 1.2 meters, outer layer diameter 3 meters), suspended in the center of the classroom, 2.5 meters above the ground. Voiceprint Collector: Four directional microphones embedded in the edges of desks (two per row), used to collect individual speech (answering questions) and writing friction sounds. Functional Design: Sound Source Separation: Beamforming technology distinguishes between teacher lectures, student discussions, and environmental noise (page turning sounds), supporting voiceprint ID binding (identifying specific student's speech). Anomaly Detection: Identifies sudden high-decibel events (falling or impact sounds), triggering the PTZ camera to turn urgently.

[0064] 1.3 Environmental Sensor Network Layer Equipment Deployment: IoT Nodes: One wireless sensor node is deployed every 3㎡ (approximately 15-20 per standard classroom), fixed to the side of the desk or wall at a height of 1.2 meters (human sitting breathing area). Sensor Types: Air Quality: Non-dispersive infrared (NDIR) CO2 sensor (range 0-5000ppm, error ±50ppm), PM2.5 laser sensor. Physical Environment: Digital temperature and humidity sensor (accuracy ±0.3℃), illuminance sensor (range 0-1000Lux). Energy Monitoring: Smart sockets collect power consumption data from projectors / air conditioners, etc. Functional Design: Dynamic Sampling: Data is uploaded every 10 seconds under normal conditions; when CO2 concentration >800ppm or temperature and humidity change rate >5% / minute, it switches to 1-second high-frequency sampling. Self-Calibration Mechanism: Benchmark calibration is initiated at 2 AM daily (synchronizing all node data based on the sensors in the podium area).

[0065] Step 102: Use the improved Kalman filtering algorithm to eliminate sensor noise in the multi-source heterogeneous data, and perform spatiotemporal alignment on the noise-eliminated data to obtain aligned multi-source heterogeneous data.

[0066] Specifically, in this embodiment, an improved Kalman filtering algorithm is used to eliminate sensor noise in the multi-source heterogeneous data, an adaptive noise covariance matrix is ​​established for different sensor characteristics, and the filtering parameters are dynamically adjusted.

[0067] By aligning the timestamps of image, sound, and temperature / humidity data, and achieving cross-modal data association through extended state vectors, audio energy fluctuations are analyzed synchronously with student actions in videos to obtain aligned multi-source heterogeneous data.

[0068] Based on the PTP precise time protocol, devices in the noise-reduced multi-source heterogeneous data are always synchronized. The network latency difference is eliminated by hardware timestamps to obtain time-aligned data.

[0069] The physical locations of the sensor nodes in the time-aligned data are mapped to the camera coordinate system. The correspondence between the desk and the image pixels is established through the QR code calibration board to obtain environmental mapping data. The tracking path of the PTZ camera is dynamically corrected using the sound source localization results to obtain visual association data.

[0070] By tracking key points in the skeleton, the combined actions of standing and talking and dozing off at the table are identified, and the duration is determined by combining time series analysis; a comprehensive scoring model is constructed by analyzing facial orientation, gaze focus and micro-expressions.

[0071] 2.1 Multi-source data spatiotemporal alignment and time synchronization: PTP (Precise Time Protocol) is used to achieve microsecond-level clock synchronization across all devices, and hardware timestamps are used to eliminate network latency differences. Spatial calibration: Visual-environment data mapping: The physical locations of sensor nodes are mapped to the camera coordinate system. For example, a QR code calibration board is used to establish the correspondence between desks and image pixels. Acoustic-visual association: The tracking path of the PTZ camera is dynamically corrected using the sound source localization results (teacher's movement trajectory).

[0072] 2.2 Data Cleaning and Feature Extraction & Noise Suppression: Visual Data: Based on motion optical flow analysis, effective actions (raising a hand) and environmental interference (curtain swaying) are distinguished. Acoustic Data: Steady-state noise (air conditioner humming) is eliminated using spectral subtraction, while transient features (coughing) are preserved. Behavioral Semantic Analysis: Group Behavior Recognition: Composite actions such as "standing discussion" and "dozing off at the table" are identified through skeleton keypoint tracking (17 joints), and the duration is determined by combining time series analysis (continuous head-down posture > 2 minutes is marked as distraction). Individual Attention Calculation: A comprehensive scoring model is constructed by analyzing facial orientation (angle deviation from the blackboard), gaze focus (coordinates of the intersection with the projection screen), and micro-expressions (frequency of frowning).

[0073] 2.3 Teaching Context Reasoning Engine Scene Classification Model: Input: Timetable information associated with the current timestamp ("Physics Experiment Class"), equipment status (experimental equipment is powered on), group behavior pattern (multiple people gathered around the operating table). Output: Dynamically label the teaching stage as "theoretical explanation", "group experiment", "summary and Q&A", etc., and load the corresponding monitoring strategy template. Knowledge Graph Query: 87 predefined teaching events ("teacher writes key points on the blackboard", "student group dispute"). When relevant features are detected, event chain reasoning is triggered (detecting 3 students leaving their seats at the same time → querying the timetable to see if it is an activity class → if it is not an activity class, mark it as abnormal).

[0074] Step 103: Establish a CRNN three-layer convolutional recurrent neural network model, input the aligned multi-source heterogeneous data into the CRNN three-layer convolutional recurrent neural network model for recognition, and obtain the classroom teaching environment status;

[0075] Specifically, in this embodiment, the first layer of the CRNN three-layer convolutional recurrent neural network model is used to extract image edge features, the second layer is used to identify dynamic behavior, and the third layer is used to fuse temporal data.

[0076] A bidirectional LSTM network is introduced into the recurrent layer of the model to process the temporal dependencies of sound signals and capture the patterns of environmental state changes.

[0077] The CRNN three-layer convolutional recurrent neural network model was trained using a historical classroom environment dataset in the database. The importance of different modalities was weighted through an attention mechanism to obtain the trained CRNN three-layer convolutional recurrent neural network model.

[0078] The classroom teaching environment status includes at least the teaching behavior recognition status and the environmental anomaly detection status;

[0079] The quality of teacher-student interaction is judged based on the activity level of group discussions, and the student's focus is analyzed through facial orientation and eye focus to obtain the teaching behavior recognition status.

[0080] The system identifies abnormal environmental conditions by detecting equipment malfunctions, safety hazards, and emergencies in the classroom environment.

[0081] 1. Model Architecture and Core Design CRNN is a hybrid model integrating Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN, LSTM / GRU), suitable for processing spatiotemporal sequence data. In this technical solution, the "three layers" typically refer to: First Layer: Convolutional Layer (CNN): Extracts spatial / local features from multi-source heterogeneous data. For environmental image data: Visual features (student posture, classroom item placement) are extracted through convolutional kernels (3×3, 5×5). For environmental sound data: Audio is first converted into Mel-Spectrogram or MFCC features, and then acoustic features (speech sound, noise intensity) are extracted through convolution. For temperature and humidity data: Although it is scalar data, it can be unified with image / sound features through dimensionality expansion (increased to 1×1×2), or a separate lightweight convolutional layer can be designed to handle temporal trends. Second Layer: Recurrent Layer (RNN / LSTM / GRU): Captures the temporal dependencies of the data (i.e., "temporal features"). Input: Feature sequences output by the convolutional layer (image frame sequences, sound segment sequences, temperature and humidity time series). Functionality: By hiding state propagation, it models the dynamic changes of classroom environment data over time (e.g., analyzing the fluctuation of sound decibels over a period of time to judge classroom activity, or predicting comfort risk based on temperature and humidity trends). Third Layer: Fully Connected Layer (FC) + Output Layer: Fuses multimodal features and performs classification / regression. Fully Connected Layer: Concatenates or weights the features (spatial features, temporal features) of convolutional and recursive layers to form a global representation. Output Layer: Based on the task design, it outputs classroom state categories ("normal teaching," "abnormally noisy," "equipment malfunction") via softmax, or outputs risk scores for environmental parameters via regression.

[0082] 2. The multimodal data fusion strategy involves three types of heterogeneous data: image, sound, and temperature / humidity. Cross-modal fusion needs to be implemented in the model: Early Fusion: In the early stages of the input layer or convolutional layer, different modal data are converted into feature vectors of a unified dimension and then concatenated (merging image features, sound spectral features, and temperature / humidity scalars into a single feature vector). Late Fusion: Each modal data is processed through independent CNN+RNN branches, and features are merged again at the fully connected layer stage (weighted summation or concatenation of the outputs from the image branch, sound branch, and temperature / humidity branch). Hybrid Fusion: Combining the above two methods, for example, using early fusion for images and sound (both belonging to sensory data), and late fusion for temperature / humidity data (environmental parameters are processed independently). II. Model Building Steps 1. Data Preprocessing and Spatiotemporal Alignment (Preliminary Steps) Noise Reduction: Improved Kalman filtering algorithms (extended Kalman filter, unscented Kalman filter) are used to denoise the original sensor data (especially time-series data such as temperature / humidity and sound decibels) to reduce measurement errors. Spatiotemporal alignment: Temporal alignment: Unifying timestamps for multi-source data (based on video frame rate, interpolating or downsampling audio and temperature / humidity data to ensure each frame corresponds to audio / temperature / humidity data at the same time point). Spatial alignment: Uniformly scaling image data (224×224), converting audio data into fixed-length segments (1 second / segment), and aggregating temperature / humidity data into statistics (mean, variance) by time window (every 10 seconds).

[0083] 3. Network Architecture Design and Convolutional Layer Configuration: Image Branch: Adopts the classic CNN architecture (a simplified version of ResNet and VGG), containing 2-3 layers of convolutions + pooling to extract visual features (student attention state, teacher position). Sound Branch: 1D or 2D convolutions (for spectrograms) to capture frequency-time domain features (keywords in speech, noise peaks). Temperature and Humidity Branch: 1D convolutional layers to extract temporal fluctuation features (sudden temperature rise, abnormal humidity fluctuations). Recursive Layer Selection: If the data has strong temporal dependencies (continuous classroom sound, temperature and humidity changes), use LSTM / GRU (to avoid the RNN gradient vanishing problem); if computational efficiency is prioritized, a simple RNN or a temporal convolutional network (TCN) can be used. Number of Recursive Layers: Usually 1-2 layers, with the number of hidden units per layer set according to data complexity (128, 256). Fusion Layer Design: After the recursive layers, multimodal features are concatenated using a fully connected layer (concat operation), and a Dropout layer is added to prevent overfitting.

[0084] 4. Training Loss Function Configuration: Select the loss function based on the output task. Use cross-entropy loss for classification tasks and mean squared error (MSE) for regression tasks. Optimizer: Adam, SGD, etc. The learning rate can be adjusted using cosine annealing or step-by-step decay. Data Augmentation: Flip, crop, and adjust the brightness of image data; add Gaussian noise and time stretch to sound data; enhance generalization of temperature and humidity data by generating simulated sequences. Training Process: Use a labeled classroom environment dataset (normal / abnormal state labels), input it into the model in batches, update parameters through backpropagation, and monitor overfitting on the validation set. III. Model Utilization and Application Process 1. Input and Inference Input Data: Preprocess the spatiotemporally aligned multi-source data (image frames, sound clips, temperature and humidity values) into a format acceptable to the model (normalization, dimensionality adaptation). Forward propagation: Convolutional layers extract features from various modalities (students raising their hands in images, applause in audio); recursive layers process feature sequences and capture time dependencies (temperature and humidity exceeding thresholds for 5 consecutive minutes, or sound decibels continuously increasing within 10 seconds); fully connected layers fuse features and output classroom status ("students are highly distracted," "equipment is abnormally overheating," "environment is stuffy").

[0085] 5. Classroom Status Recognition and Output Status Classification: Output discrete categories ("Normal Teaching", "Abnormal Noise", "Uncomfortable Environment") or continuous values ​​(classroom activity score, risk index). Multimodal Collaborative Decision Making: For example, if students are frequently standing in an image (visual feature) + there is noise in the sound (acoustic feature) + the temperature and humidity are normal (environmental parameters), the overall judgment is "active classroom interaction"; if the temperature and humidity are also excessive, it is judged as "high workload risk".

[0086] 6. Risk Warning and Environmental Control: Risk Warning: Triggers an alarm based on the output status (based on a threshold; when the probability of an "abnormal state" exceeds 80%, a notification is sent to the teacher or administrator). Adaptive Control: If "humid environment" is detected (temperature and humidity data + image characteristics of students sitting sluggishly), the air conditioner is automatically activated to adjust the temperature; if "abnormal equipment noise" is detected (abnormal sound spectrum + smoke emitted from the equipment captured by the camera), the equipment power-off protection is triggered and an alarm is sounded.

[0087] Step 104: Conduct classroom risk warning and adaptive classroom environment control based on the classroom teaching environment status.

[0088] Specifically, in this embodiment, an audible and visual alarm is triggered for fire or sudden loud noise events, and information is pushed to the administrator terminal;

[0089] Based on the duration of temperature and humidity deviations from the comfort zone and the proportion of students distracted, a three-level warning system (yellow / orange / red) is generated to prompt teachers to adjust their teaching strategies.

[0090] The system uses a PID algorithm to control the air conditioning and lighting systems. When it detects a drop in student concentration, it pushes interactive courseware or switches the teaching scene.

[0091] 3.1 Anomaly Detection and Tiered Response Multi-Threshold Trigger Mechanism: Level 1 Warning (Low Risk): When environmental parameters deviate from the comfort zone (illuminance <300Lux) but teaching behavior is normal, only logs are recorded. Level 2 Intervention (Medium Risk): When a local CO2 concentration >1200ppm is detected and the frequency of student yawning increases, the fresh air system is automatically activated and a notification is pushed to the teacher's tablet. Level 3 Emergency Response (High Risk): When fire smoke or violent physical conflict is detected, an emergency broadcast is directly triggered and the campus security system is activated.

[0092] 3.2 Adaptive Environmental Control Multi-Objective Optimization Strategy: Priority Rule: Control of teaching equipment (projector) takes precedence over environmental equipment (air conditioner) to avoid screen shaking caused by adjusting air conditioner fan speed. Conflict Resolution: When it is necessary to simultaneously lower the temperature (due to increased student activity) and increase the illuminance (due to cloudy weather), a sequential decision of "adjusting light first, then adjusting temperature" is adopted to avoid student discomfort caused by frequent environmental changes. Implicit Adjustment Technology: Light adjustment uses a gradual algorithm (change <50 Lux per minute), and temperature adjustment fan speed is limited to <3 m / s to ensure imperceptible adjustment for the human body.

[0093] 3.3 Teaching Aid Feedback Teacher-Side AR Interface: Visual Presentation: A lightweight AR glasses layer is overlaid with information to display real-time data (average class focus rate 72%, good air quality index) in the lower right corner of the teacher's field of vision. Interaction Design: Supports iris gaze triggering detailed analysis (after gazing at the "interaction balance" index for 3 seconds, a distribution chart of speaking time for each group is displayed). Student-Side Invisible Feedback: By adjusting the LED light color temperature (switching from 6500K cool white to 4000K warm white), focus is subtly improved, avoiding direct reminders that could cause psychological pressure.

[0094] Its beneficial effects lie in the construction of a three-dimensional monitoring system through a multi-source heterogeneous data acquisition mechanism. The solution integrates multi-modal data such as environmental images, sound, temperature, and humidity, overcoming the limitations of single-sensor monitoring and comprehensively capturing classroom dynamics. It can capture students' attention states through image recognition, identify abnormal sounds through sound analysis, and assess environmental comfort in real time by combining temperature and humidity data, providing comprehensive data support for classroom monitoring. Secondly, the improved Kalman filtering algorithm and spatiotemporal alignment technology ensure data quality. Addressing sensor noise issues, the signal-to-noise ratio is improved by optimizing the filtering model, and spatiotemporal alignment processing eliminates time differences and spatial deviations in multi-source data, ensuring that images, sound, and environmental parameters are fused in a unified spatiotemporal coordinate system, laying a reliable foundation for subsequent intelligent analysis. The CRNN model and adaptive control form a closed-loop management system. Utilizing the spatiotemporal feature extraction capabilities of convolutional recurrent neural networks, the system accurately identifies classroom environmental conditions and triggers risk warnings and environmental controls in real time. This mechanism not only proactively prevents sudden risks in the classroom but also automatically adjusts intelligent air conditioning and lighting systems to create a safe and comfortable teaching environment, which is conducive to improving classroom efficiency and protecting the health of teachers and students.

[0095] Please see Figure 2 In the intelligent classroom environment monitoring method based on data analysis, acquiring multi-source heterogeneous data from the classroom environment, including at least environmental image data, environmental sound data, and environmental temperature and humidity data, includes the following steps:

[0096] Step 201: Acquire environmental image data in the classroom environment through an image sensor. The environmental image data includes at least color images, depth information, and backlighting.

[0097] Step 202: Use a channel ring microphone array to acquire individual student speech data, teacher's lecturing voice, student discussion voice, environmental noise and writing friction sound in the classroom environment to obtain environmental sound data;

[0098] Step 203: Obtain temperature and humidity data in the classroom environment using temperature and humidity sensors to obtain environmental temperature and humidity data.

[0099] The above describes embodiments of the intelligent classroom environment monitoring method based on data analysis of the present invention. Please refer to [link / reference]. Figure 3 In the data-driven intelligent classroom environment monitoring system, the system includes the following modules:

[0100] The environmental data acquisition module is used to acquire multi-source heterogeneous data in the classroom environment. The multi-source heterogeneous data includes at least environmental image data, environmental sound data, and environmental temperature and humidity data.

[0101] The data alignment processing module is used to eliminate sensor noise in the multi-source heterogeneous data using an improved Kalman filtering algorithm, and to perform spatiotemporal alignment on the noise-eliminated data to obtain aligned multi-source heterogeneous data.

[0102] The environmental state recognition module is used to establish a CRNN three-layer convolutional recurrent neural network model, input the aligned multi-source heterogeneous data into the CRNN three-layer convolutional recurrent neural network model for recognition, and obtain the classroom teaching environment state;

[0103] The classroom environment monitoring module is used to provide early warnings of classroom risks and adaptive classroom environment control based on the status of the classroom teaching environment.

[0104] Please see Figure 4 This is a schematic diagram of the CRNN three-layer recurrent convolutional neural network model in the intelligent monitoring method of classroom environment based on data analysis.

[0105] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A data-driven intelligent monitoring method for classroom environments, characterized in that: The intelligent classroom environment monitoring method includes the following steps: Acquire multi-source heterogeneous data from the classroom environment, wherein the multi-source heterogeneous data includes at least environmental image data, environmental sound data, and environmental temperature and humidity data; The sensor noise of the multi-source heterogeneous data is eliminated by using an improved Kalman filtering algorithm. The data after noise elimination is then spatiotemporally aligned to obtain aligned multi-source heterogeneous data. A three-layer convolutional recurrent neural network (CRNN) model is established. The aligned multi-source heterogeneous data is input into the CRNN model for recognition to obtain the classroom teaching environment status. The classroom teaching environment status includes at least the teaching behavior recognition status and the environmental anomaly detection status. The quality of teacher-student interaction is judged based on the activity level of group discussions. The student focus is analyzed through facial orientation and eye focus to obtain the teaching behavior recognition status. The environmental anomaly detection status is identified through equipment failures, safety hazards, and emergencies in the classroom environment. Based on the described classroom teaching environment status, conduct classroom risk warnings and adaptive classroom environment adjustments; The step of using an improved Kalman filtering algorithm to eliminate sensor noise in the multi-source heterogeneous data, and then performing spatiotemporal alignment on the noise-reduced data to obtain aligned multi-source heterogeneous data, further includes: Based on the PTP precise time protocol, devices in the noise-reduced multi-source heterogeneous data are always synchronized. The network latency difference is eliminated by hardware timestamps to obtain time-aligned data. The physical locations of the sensor nodes in the time-aligned data are mapped to the camera coordinate system. The correspondence between the desk and the image pixels is established through the QR code calibration board to obtain environmental mapping data. The tracking path of the PTZ camera is dynamically corrected using the sound source localization results to obtain visual association data. By tracking key points in the skeleton, the combined actions of standing and talking and dozing off at the table are identified, and the duration is determined by combining time series analysis; a comprehensive scoring model is constructed by analyzing facial orientation, gaze focus and micro-expressions.

2. The intelligent classroom environment monitoring method based on data analysis as described in claim 1, characterized in that, The acquisition of multi-source heterogeneous data in the classroom environment, including at least environmental image data, environmental sound data, and environmental temperature and humidity data, includes: Environmental image data of the classroom environment is acquired through an image sensor, and the environmental image data includes at least color images, depth information, and backlit scenes; A channel ring microphone array was used to acquire individual student speech data, teacher lecturing voice, student discussion voice, environmental noise, and writing friction sound in the classroom environment to obtain environmental sound data. Temperature and humidity data in the classroom environment are obtained by using temperature and humidity sensors.

3. The intelligent classroom environment monitoring method based on data analysis as described in claim 1, characterized in that, The process of eliminating sensor noise from the multi-source heterogeneous data using an improved Kalman filtering algorithm, and then performing spatiotemporal alignment on the noise-removed data to obtain aligned multi-source heterogeneous data includes: An improved Kalman filtering algorithm is used to eliminate sensor noise in the multi-source heterogeneous data. An adaptive noise covariance matrix is ​​established for different sensor characteristics, and the filtering parameters are dynamically adjusted. By aligning the timestamps of image, sound, and temperature / humidity data, and achieving cross-modal data association through extended state vectors, audio energy fluctuations are analyzed synchronously with student actions in videos to obtain aligned multi-source heterogeneous data.

4. The intelligent classroom environment monitoring method based on data analysis as described in claim 1, characterized in that, The establishment of the CRNN three-layer convolutional recurrent neural network model includes: The first layer of the CRNN three-layer convolutional recurrent neural network model is used to extract image edge features, the second layer is used to identify dynamic behavior, and the third layer is used to fuse temporal data. A bidirectional LSTM network is introduced into the recurrent layer of the model to process the temporal dependencies of sound signals and capture the patterns of environmental state changes. The CRNN three-layer convolutional recurrent neural network model was trained using a historical classroom environment dataset in the database. The importance of different modalities was weighted through an attention mechanism to obtain the trained CRNN three-layer convolutional recurrent neural network model.

5. The intelligent classroom environment monitoring method based on data analysis as described in claim 1, characterized in that, The method of providing classroom risk warnings and adaptive classroom environment control based on the classroom teaching environment status includes: The system triggers audible and visual alarms in response to fires or sudden loud noise events, and pushes information to the administrator's terminal. Based on the duration of temperature and humidity deviations from the comfort zone and the proportion of students distracted, a three-level warning system (yellow / orange / red) is generated to prompt teachers to adjust their teaching strategies. The system uses a PID algorithm to control the air conditioning and lighting systems. When it detects a drop in student concentration, it pushes interactive courseware or switches the teaching scene.

6. A data-analysis-based intelligent classroom environment monitoring system, used to execute the data-analysis-based intelligent classroom environment monitoring method as described in claim 1, characterized in that, Includes the following modules: The environmental data acquisition module is used to acquire multi-source heterogeneous data in the classroom environment. The multi-source heterogeneous data includes at least environmental image data, environmental sound data, and environmental temperature and humidity data. The data alignment processing module is used to eliminate sensor noise in the multi-source heterogeneous data using an improved Kalman filtering algorithm, and to perform spatiotemporal alignment on the noise-eliminated data to obtain aligned multi-source heterogeneous data. An environmental state recognition module is used to establish a CRNN three-layer convolutional recurrent neural network model. The aligned multi-source heterogeneous data is input into the CRNN three-layer convolutional recurrent neural network model for recognition to obtain the classroom teaching environment state. The classroom teaching environment state includes at least the teaching behavior recognition state and the environmental anomaly detection state. The quality of teacher-student interaction is judged based on the activity level of group discussions, and the student's focus is analyzed through facial orientation and eye focus to obtain the teaching behavior recognition state. Identify abnormal environmental conditions by detecting equipment malfunctions, safety hazards, and emergencies in the classroom environment; The classroom environment monitoring module is used to provide early warnings of classroom risks and adaptive classroom environment control based on the status of the classroom teaching environment.

7. The intelligent classroom environment monitoring system based on data analysis as described in claim 6, characterized in that, The environmental data acquisition module includes the following sub-modules: The image data acquisition submodule is used to acquire environmental image data in the classroom environment through an image sensor. The environmental image data includes at least color images, depth information, and backlighting scenes. The sound data acquisition submodule is used to acquire individual student speech data, teacher lecturing voice, student discussion voice, environmental noise and writing friction sound in the classroom environment using a channel ring microphone array, to obtain environmental sound data. The temperature and humidity data acquisition submodule is used to acquire temperature and humidity data in the classroom environment through temperature and humidity sensors to obtain environmental temperature and humidity data.

8. The intelligent classroom environment monitoring system based on data analysis as described in claim 6, characterized in that, The data alignment processing module includes the following sub-modules: The time alignment submodule is used to synchronize devices in noise-reduced multi-source heterogeneous data based on the PTP precise time protocol. It eliminates network latency differences through hardware timestamps to obtain time-aligned data. The spatial alignment submodule is used to map the physical positions of sensor nodes in the time alignment data to the camera coordinate system, establish the correspondence between the desk and image pixels through the QR code calibration board to obtain environmental mapping data, and dynamically correct the tracking path of the PTZ camera using the sound source localization results to obtain visual association data. The data processing and analysis submodule is used to identify the combined actions of standing and talking and dozing off by tracking the skeleton key points, and to determine the duration by combining time series analysis. It also analyzes facial orientation, gaze focus and micro-expressions to build a comprehensive scoring model.

Images