Classroom student behavior identification analysis method and system based on edge calculation

Abstract

The invention belongs to the technical field of behavior recognition, and particularly discloses a classroom student behavior recognition analysis method and system based on edge computing, and the method comprises the steps: optimizing a YOLOv12 model, obtaining an optimized YOLOv12 model, and deploying the optimized YOLOv12 model on an edge computing platform; inputting video data to be identified and analyzed into the optimized YOLOv12 model to obtain a behavior identification result; and analyzing the behavior recognition result through a large language model deployed on the edge computing platform to obtain an analysis result. According to the method, the behavior identification precision can be improved, the dependence on cloud computing is reduced, the network bandwidth consumption is reduced, the data security is improved, and a deep teaching reference beyond manual observation is provided for teachers.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of behavior recognition, and more particularly, relates to a classroom student behavior recognition analysis method and system based on edge computing. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, especially deep learning technology, computer vision has been widely applied in various fields. The classroom behavior recognition system, as a typical application of computer vision, aims to automatically recognize the behaviors of students and teachers by analyzing video data in the classroom environment, and then provide effective feedback for the teaching process. Such a system not only can monitor the behavior patterns of students in real time, but also can analyze the teaching behaviors of teachers, so as to optimize the teaching methods and improve the efficiency of classroom management.

[0003] However, the existing classroom behavior recognition methods still face many challenges. On the one hand, most systems rely on cloud servers for centralized processing, which not only requires the transmission of massive video data, resulting in huge network bandwidth consumption and transmission delay, but also makes it difficult to meet the stringent demand for real-time feedback in classroom teaching. Moreover, uploading sensitive classroom monitoring videos to the cloud also raises serious concerns about data privacy leakage. On the other hand, the classroom scene has complex characteristics such as personnel concentration, serious mutual occlusion, and large target size variation (such as small targets of students in the back row). Existing general detection models often fail to extract features with strong robustness, resulting in frequent missed detection and false detection when dealing with dense occlusion or subtle actions, low recognition accuracy, and difficulty in meeting the requirements of precise teaching analysis. SUMMARY

[0004] In view of the defects of the prior art, the purpose of the present application is to provide a classroom student behavior recognition analysis method and system based on edge computing, which aims to solve the problem of low recognition accuracy caused by the difficulty of existing general detection models in extracting features with strong robustness.

[0005] To achieve the above-mentioned purpose, in a first aspect, the present application provides a classroom student behavior recognition analysis method based on edge computing, comprising: optimizing a YOLOv12 model to obtain an optimized YOLOv12 model, and deploying the optimized YOLOv12 model on an edge computing platform; inputting the video data to be recognized and analyzed into the optimized YOLOv12 model to obtain a behavior recognition result; analyzing the behavior recognition result by a large language model deployed on the edge computing platform to obtain an analysis result.

[0006] The application optimizes the YOLOv12 model, enhances the feature extraction capability of multi-scale and irregular behavior targets in the classroom scene, can overcome the problem that the existing general detection model is difficult to extract features with strong robustness, and then improve the recognition accuracy. By deploying the deep learning model on the edge computing platform, the dependence on cloud computing can be reduced, the network bandwidth consumption can be reduced, and the data security can be improved. By inputting the behavior recognition result into the large language model, data analysis can be automatically completed, cross-time dimension analysis and analysis of the implicit correlation between behaviors can be realized, and a deep teaching reference beyond manual observation can be provided for teachers.

[0007] According to the classroom student behavior recognition analysis method based on edge computing provided by the application, the YOLOv12 model is optimized, which comprises: A two-dimensional deformable attention module (Deformable Attention 2D) is embedded in the detection head pre-feature layer or feature fusion network of the YOLOv12 model.

[0008] By embedding the two-dimensional deformable attention module in the detection head pre-feature layer or feature fusion network of the YOLOv12 model, the receptive field is adaptively adjusted by learning the offset of the sampling point, so as to enhance the feature extraction capability of multi-scale and irregular behavior targets in the classroom scene.

[0009] According to the classroom student behavior recognition analysis method based on edge computing provided by the application, the YOLOv12 model is optimized, which comprises: The backbone network of the YOLOv12 model is replaced by MobileNetV4.

[0010] The backbone network of the YOLOv12 model is replaced by MobileNetV4, and the inverted residual structure and linear bottleneck design of MobileNetV4 are used to reduce the network parameter quantity and calculation quantity under the premise of ensuring the detection accuracy of the model.

[0011] According to the classroom student behavior recognition analysis method based on edge computing provided by the application, the YOLOv12 model is optimized, which comprises: The YOLOv12 model is subjected to network pruning and quantization operation.

[0012] The YOLOv12 model is subjected to network pruning and quantization operation, so as to reduce the deployment load of the model on the edge computing platform.

[0013] According to the classroom student behavior recognition analysis method based on edge computing provided by the application, the YOLOv12 model is optimized, which comprises: Accelerate the inference process of YOLOv12 models through TensorRT and Compute Unified Device Architecture (CUDA).

[0014] According to the edge computing-based classroom student behavior recognition and analysis method provided in this application, the method further includes: The behavior recognition results and the analysis results are visualized.

[0015] Secondly, this application provides a classroom student behavior recognition and analysis system based on edge computing, comprising: The optimization module is used to optimize the YOLOv12 model, obtain the optimized YOLOv12 model, and deploy the optimized YOLOv12 model on the edge computing platform. The recognition module is used to input the video data to be recognized and analyzed into the optimized YOLOv12 model to obtain the behavior recognition results. The analysis module is used to analyze the behavior recognition results using a large language model deployed on the edge computing platform to obtain analysis results.

[0016] Thirdly, this application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the edge computing-based classroom student behavior recognition and analysis method described in the first aspect or any possible implementation thereof.

[0017] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the edge computing-based classroom student behavior recognition and analysis method described in the first aspect or any possible implementation of the first aspect.

[0018] Fifthly, this application provides a computer program product that, when run on a processor, causes the processor to execute the edge computing-based classroom student behavior recognition and analysis method described in the first aspect or any possible implementation of the first aspect.

[0019] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0020] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art: This application improves and optimizes the YOLOv12 model to enhance its feature extraction capabilities for multi-scale and irregular behavioral targets in classroom scenarios. This overcomes the problem that existing general detection models struggle to extract robust features, thereby improving recognition accuracy. By deploying deep learning models on edge computing platforms, reliance on cloud computing can be reduced, network bandwidth consumption can be decreased, and data security can be improved. By inputting the behavior recognition results into a large language model, data analysis can be automatically completed, enabling cross-time dimension analysis and analysis of implicit relationships between behaviors, providing teachers with in-depth teaching references that go beyond manual observation. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the classroom student behavior recognition and analysis method based on edge computing provided in the embodiments of this application; Figure 2 This is a flowchart illustrating the edge deployment and application of a large language model provided in an embodiment of this application. Figure 3 This is a flowchart illustrating the edge deployment process of the visual model provided in this application embodiment; Figure 4 This is an overall framework diagram of classroom behavior recognition and analysis provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the classroom student behavior recognition and analysis system based on edge computing provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0024] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0025] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0026] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0027] Next, combined Figures 1-4 This paper introduces the classroom student behavior recognition and analysis method based on edge computing provided in the embodiments of this application.

[0028] Figure 1 This is a flowchart illustrating the classroom student behavior recognition and analysis method based on edge computing provided in an embodiment of this application. Figure 1 As shown, the method includes the following steps: Step S1: Optimize the YOLOv12 model to obtain the optimized YOLOv12 model, and deploy the optimized YOLOv12 model on the edge computing platform; This application optimizes and improves the YOLOv12 object detection model by introducing a deformable attention mechanism to enhance its feature extraction capabilities. This is particularly beneficial in complex and dynamic classroom environments, enabling more accurate identification of student and teacher behaviors. The advantage of the YOLOv12 network lies in its ability to perform multi-object detection simultaneously, analyzing various behaviors in video data in real time, such as students raising their hands, looking up, and whispering, as well as teachers' blackboard writing and lecturing. In this way, the system not only improves detection accuracy but also reduces response time, ensuring that classroom behaviors are identified and responded to promptly.

[0029] Optionally, the optimization of the YOLOv12 model in this application can be an algorithmic optimization and a deployment optimization, with the aim of enhancing its feature extraction capability for multi-scale and irregular behavioral targets in classroom scenarios and ensuring that the model can run efficiently on edge devices.

[0030] Step S2: Input the video data to be identified and analyzed into the optimized YOLOv12 model to obtain the behavior recognition results; Optionally, video data of the classroom scene is collected by camera sensors installed in the classroom to obtain behavioral image information of students and teachers. The collected video data is then transmitted to the YOLOv12 model deployed on the edge computing platform, and the behavioral recognition results are obtained through the YOLOv12 model, including students' head-up, reading and writing, sleeping, raising hands, whispering, standing up, playing with mobile phones, group discussion behavior, and teachers' blackboard writing, PPT presentation, student guidance, and classroom patrol behavior.

[0031] Alternatively, the edge computing platform may be the NVIDIA Jetson edge computing platform or other edge computing platforms. In one embodiment of this application, the NVIDIA Jetson edge computing platform is deployed locally in the classroom.

[0032] This application pushes data processing directly to the edge computing platform to reduce data transmission latency and bandwidth consumption, while improving data security. The edge computing platform can reduce its dependence on cloud computing when processing data, ensuring system response speed and processing capacity, and is especially suitable for use in environments with limited computing resources.

[0033] Step S3: Analyze the behavior recognition results using a large language model deployed on an edge computing platform to obtain the analysis results.

[0034] The behavior recognition model primarily detects and records student classroom behavior. However, analyzing student classroom behavior requires the involvement of a larger model. This larger model analyzes the inference results of the smaller model to generate specific classroom behavior recognition reports.

[0035] The integration of existing classroom behavior recognition systems with large language models has significant limitations: most solutions deploy large language models in the cloud, requiring sensitive classroom behavior data to be uploaded for analysis, posing a risk of privacy breaches; simultaneously, cloud analysis suffers from data transmission delays, hindering real-time feedback; furthermore, general-purpose large language models have high computational complexity, making direct deployment on resource-constrained edge devices difficult, thus preventing the completion of the "behavior recognition-deep analysis-teaching feedback" closed loop locally. Therefore, this application introduces a lightweight large model and deploys it on an edge computing platform for edge-side semantic analysis. Based on high-precision recognition results, it performs deep contextual understanding, trend insights, and intelligent report generation for students' learning status and teachers' teaching modes, providing semantic-level feedback that goes beyond simple data statistics, enabling local deep analysis of classroom behavior.

[0036] Figure 2 This is a flowchart illustrating the edge deployment and application of a large language model provided in this application embodiment, such as... Figure 2As shown, in the large model selection phase, mainstream open-source models (such as Qwen, Llama, and distilled Deepseek) are prioritized, with parameter counts controlled between 3B and 7B to adapt to edge deployment requirements. Performance evaluation is used to select the optimal model for subsequent deployment. In the large model deployment phase, commonly used inference frameworks (such as vLLM, SGLang, and TensorRT-LLM) are used to evaluate the selected models, ultimately selecting the framework with the best inference performance for deployment. These inference frameworks incorporate multiple optimization mechanisms, including Flash Attention for optimized attention calculation, Paged Attention for optimized memory usage, and continuous batch processing for optimized multi-batch inference, effectively improving the inference efficiency of large models. In the system construction phase, a collaborative pipeline between the large model and the visual model is built, reading historical behavioral data from the database across time dimensions to automatically generate comparative reports for individual students (e.g., monthly attention changes) or the entire class (e.g., semester interaction frequency trends), intuitively presenting the dynamic changes in learning status. Beyond surface-level behavioral statistics (such as the number of times hands are raised), large models can also analyze implicit correlations between behaviors, such as the correlation between "high frequency of looking down" and "weak grasp of knowledge points," providing teachers with in-depth teaching references that go beyond manual observation. Furthermore, by combining individual student behavioral data (such as a student only focusing when asking questions) with overall class data, differentiated teaching suggestions can be generated.

[0037] Optionally, all behavior recognition results of the YOLOv12 model can be stored in a database for subsequent analysis and querying. This not only helps teachers review classroom behavior but also provides targeted teaching improvement suggestions through the analysis of historical data. Through database management, teachers can fully understand the changing trends of student behavior and adjust their teaching strategies accordingly to improve classroom management effectiveness.

[0038] Optionally, in the integration of the large model with the behavior recognition system, an interface for calling the large model can be written to enable interaction with the behavior recognition model and the database: the data input interface obtains real-time behavior data from edge devices and reads historical behavior data from the database. The two types of data are then organized into structured text according to a preset format (e.g., "Real-time data: Classroom number 123, time 20250825, negative behavior percentage 15%; Historical data: percentage of negative behavior in the same classroom once 10%), which is then used as input to the large model; the result output interface receives the analysis report generated by the large model, parses it according to the format, stores it in the database, and simultaneously pushes it to the report viewing interface of the QT visualization system.

[0039] Optionally, regarding the logic and testing of the large model analysis, a large model analysis template can be preset, clearly defining the analysis dimensions (overall classroom trends, individual behavior tracking, and teaching suggestions) and output format. This ensures that the reports generated by the large model are structured and easy to read, tests the accuracy of the analysis results, compares whether the behavioral trends output by the large model are consistent with the actual data, verifies the rationality of the teaching suggestions, and ensures that the large model can provide valuable analytical support for teachers.

[0040] The edge computing-based classroom student behavior recognition and analysis method provided in this application improves and optimizes the YOLOv12 model, enhancing its ability to extract features from multi-scale and irregular behavioral targets in classroom scenarios. This overcomes the problem that existing general detection models struggle to extract robust features, thereby improving recognition accuracy. By deploying deep learning models on edge computing platforms, reliance on cloud computing is reduced, network bandwidth consumption is decreased, and data security is enhanced. By inputting the behavior recognition results into a large language model, data analysis can be automatically completed, enabling cross-time dimension analysis and analysis of implicit correlations between behaviors, providing teachers with in-depth teaching references that go beyond manual observation.

[0041] In some embodiments, step S1 specifically includes: Embed two-dimensional deformable attention modules in the pre-feature layer of the detection head or the feature fusion network of the YOLOv12 model.

[0042] Two-dimensional deformable attention modules are embedded in the pre-feature layer or feature fusion network of the detection head of the YOLOv12 model. By learning the offset of the sampling points, the receptive field is adaptively adjusted to enhance the feature extraction capability of multi-scale and irregular behavioral targets in classroom scenarios.

[0043] In some embodiments, step S1 specifically includes: Replace the backbone network of the YOLOv12 model with MobileNetV4.

[0044] The YOLOv12 model was lightweighted by selecting MobileNetV4 as the backbone network. By utilizing MobileNetV4's inverse residual structure and linear bottleneck design, the number of network parameters and computational cost were reduced while ensuring the model's detection accuracy, thus significantly reducing the consumption of computing resources.

[0045] In one embodiment of this application, during the model building phase, the YOLOv12 base model is loaded, its backbone network is replaced with MobileNetV4, a deformable attention module is integrated, and parameters are initialized; first, query vectors and offset predictions are generated. Assume the input feature map is... (in , , (Height, width, and number of channels are respectively). The input is processed by a 1×1 convolution. Perform linear projection to generate the query vector. :

[0046] in Unlike traditional attention mechanisms, this application does not use fixed grid points to generate key-value pairs, but instead utilizes convolutional networks to generate learnable weight matrices. After grouping Feature extraction is performed to predict the coordinate offset of the sampling point. :

[0047] in, To control the scaling factor of the offset magnitude, The function limits the offset to the interval (-1, 1) to prevent the sampling point from going out of bounds.

[0048] Secondly, feature resampling based on offsets, based on the generated offsets Calculate the coordinates of the deformed sampling grid. For each basic grid point on the feature map The corresponding sampling location is:

[0049] Using bilinear interpolation function In the original input feature map According to Sampling is performed to obtain the deformed intermediate features. :

[0050] Subsequently, bonds are generated through a linear projection layer. Sum : ,

[0051] This process enables the model to adaptively adjust its receptive field based on students' irregular postures in the classroom, rather than being limited to a regular rectangular region.

[0052] Next, Continuous Position Bias (CPB) is introduced to handle high-resolution query grids. With the deformed low-resolution bond mesh The positional relationship between them is used to introduce a continuous position offset matrix. First, calculate the relative coordinate distance between the two. Position biases are generated using log-space encoding and multilayer perceptron (MLP):

[0053] This bias term can effectively encode relative position information under non-rigid deformation.

[0054] Finally, calculate the attention weights and feature aggregation: Calculate and The dot product similarity, divided by the scaling factor. ( (for feature dimensions), superimposed positional bias Then, softmax normalization is performed to obtain the attention weight map. :

[0055] Finally, the weighted graph AND value Perform weighted aggregation and then use a linear projection matrix. Output the final feature representation :

[0056] This mechanism effectively enhances the model's feature extraction capabilities when dealing with dense occlusion of students and multi-scale behavioral targets at the edge.

[0057] During the model training phase, the processed training set data is input into the model and trained in batches. After each round, the model accuracy (mAP, recall, precision) is evaluated on the validation set, and the model with the best accuracy on the validation set is saved.

[0058] Optionally, the core of model training is to complete the training and initial optimization of the improved YOLOv12 model, laying the foundation for subsequent deployment. First, a classroom behavior dataset is constructed: based on the existing classroom behavior dataset, behavior annotations that do not conform to actual teaching scenarios are removed, and classroom videos from different subjects and time periods are collected. Video frames are extracted and labeled with 8 categories of student behavior and 4 categories of teacher behavior, divided into training, validation, and test sets in a 7:2:1 ratio. Data augmentation processing is performed on the training set, including random flipping, random cropping, color jittering, and Gaussian blur, to improve the model's generalization ability.

[0059] In some embodiments, step S1 specifically includes: Perform network pruning and quantization operations on the YOLOv12 model.

[0060] To further optimize model performance, network pruning and quantization operations were performed on the model to reduce the deployment load on edge computing platforms. Network pruning removed redundant neural network connections, reducing computational complexity; quantization converted floating-point weights into fixed-point weights, significantly reducing storage and computational requirements.

[0061] Optionally, network pruning can employ an L1 regularized pruning strategy. The absolute values ​​of the weight parameters of each convolutional layer in the YOLOv12 model are calculated, a weight threshold of 0.001 is set, redundant convolutional kernels with absolute weight values ​​less than this threshold are removed, and invalid connections and computational nodes in the network are eliminated. After pruning, the inference speed of the model is improved by more than 25% compared to before pruning, and the detection accuracy loss does not exceed 1%.

[0062] Optionally, the quantization operation can be performed using the TensorRT tool to quantize the model parameters of FP32 precision to FP16 or INT8 precision. The quantization process is calibrated based on the classroom behavior dataset to ensure that the detection accuracy of the quantized model is stable. The storage capacity of the INT8 quantized model is reduced by 70% compared to the original FP32 model, and the inference speed is increased by more than 1.8 times.

[0063] TensorRT, developed by NVIDIA, is a deep learning inference optimization tool designed to accelerate the inference process of neural networks. TensorRT supports various optimization techniques that significantly improve the execution efficiency of models on NVIDIA GPUs. Its main acceleration techniques include graph optimization, operator fusion, automatic kernel tuning, quantization, and multi-stream execution. Quantization converts FP32 floating-point data into FP16 or INT8, reducing storage and computational overhead, making it particularly suitable for embedded devices with limited computing resources. TensorRT supports parallel execution of multiple inference streams, fully utilizing the parallel computing capabilities of GPUs by processing multiple input streams simultaneously, further improving inference efficiency. Through these optimizations, TensorRT can significantly improve the inference speed of object detection models, making it possible to deploy deep learning models on embedded edge devices.

[0064] Optionally, after the YOLOv12 model is trained, the optimal model is pruned: the absolute value of the convolutional layer weights is calculated using a regularization tool, a weight threshold is set, redundant convolutional kernels are removed, and 50 rounds of fine-tuning training are performed after pruning to recover the accuracy loss caused by pruning. Finally, the fine-tuned model is quantized to FP16 or INT8 precision using the TensorRT tool to generate the quantized model file, thus completing the model training and preliminary optimization.

[0065] In some embodiments, step S1 specifically includes: Accelerate the inference process of the YOLOv12 model using TensorRT and CUDA.

[0066] Figure 3 This is a flowchart illustrating the edge deployment process of the visual model provided in this application embodiment, such as... Figure 3 As shown, given the relatively limited computing power of edge devices, the visual model needs to be optimized at the algorithm level. The core objective of the optimization is to maximize model lightweighting while ensuring inference accuracy, reserving sufficient computing power for the subsequent deployment of the large language model. The main optimization methods at the algorithm level include: attention mechanism optimization, lightweight convolution design, and lightweight modification of the backbone network. After completing the algorithm-level optimization, the optimized model needs to be deployed and accelerated: on the Jetson platform, the TensorRT inference engine is used to accelerate the model inference process. This engine can improve efficiency through a series of optimization measures such as operator fusion, multi-stream execution, and automatic kernel tuning; for the pre- and post-processing stages of the visual model that the TensorRT engine cannot cover, CUDA parallel computing technology is used for acceleration, and quantization technology is combined to further improve the model inference speed. After the model is deployed, a visualization display system is developed to present the detection results of the visual model in real time and store the results information in a local database. When it is necessary to analyze classroom information, the large language model reads the information in the database, performs semantic parsing, and generates a classroom analysis report.

[0067] Graph optimization techniques, employed by TensorRT, reduce computational load and accelerate inference by analyzing each node in the computation graph, removing redundant operations, merging similar computations, and optimizing memory layout. Operator fusion, another graph optimization technique, combines multiple independent operators (such as convolution, activation, and pooling) into a single operation to reduce memory accesses and computational overhead. This method significantly improves computational efficiency, especially for memory-intensive operators. Kernel auto-tuning involves TensorRT selecting and automatically optimizing the most suitable hardware kernel for each operator. By choosing the optimal kernel implementation based on the hardware architecture, it improves the utilization of the Graphics Processing Unit (GPU), thereby accelerating inference speed.

[0068] Optionally, TensorRT inference optimization specifically involves: using TensorRT to optimize the computation graph of the quantized YOLOv12 model, analyzing the model's computation flow and removing redundant operation nodes and repeated computation steps; fusing operators such as convolution, batch normalization, and activation functions in the model, integrating multiple independent operator operations into a single computation operator, reducing GPU memory access times and data transfer latency; and automatically performing kernel tuning based on the GPU architecture of NVIDIA Jetson edge devices (such as the Ampere architecture), selecting the optimal hardware computation kernel for each operator. The inference speed of the optimized model is more than 20 times faster than that of the unoptimized model, with the inference speed of the FP16 precision model approaching 40 times that of the unoptimized model.

[0069] However, TensorRT only optimizes the model inference part and cannot accelerate the pre-processing and post-processing parts of visual model inference. Therefore, this application introduces CUDA parallel computing to accelerate the pre- and post-processing processes. CUDA (Compute Unified Device Architecture) is a parallel computing platform and programming model launched by NVIDIA. Its core goal is to leverage the powerful computing capabilities of GPUs to accelerate the running efficiency of various computationally intensive applications. This platform provides a rich set of parallel computing frameworks that can fully utilize the massive number of GPU cores for parallel data processing, thereby efficiently accelerating various computing tasks. It should be noted that TensorRT's underlying optimization achieves model acceleration by implementing high-performance operators through CUDA. However, in the visual model inference process, pre- and post-processing operations cannot be accelerated by TensorRT. In this case, it is necessary to write corresponding operators using CUDA to leverage its parallel computing capabilities to improve the computational efficiency of the pre-processing stage.

[0070] The CUDA-accelerated pre- and post-processing are as follows: In the pre-processing stage, image pre-processing functions are written using CUDA parallel programming. For the input video frame image, an independent CUDA thread is allocated to each pixel to synchronously perform bilinear interpolation image scaling, pixel value normalization (mapping pixel values ​​to the [0,1] interval), and BGR to RGB color channel conversion operations. In the post-processing stage, CUDA multi-threading technology is used to allocate an independent thread to each model output prediction box to perform prediction box coordinate decoding (converting normalized coordinates to image pixel coordinates) and non-maximum suppression (setting the IOU threshold to 0.5 to remove redundant prediction boxes with high overlap) operations in parallel.

[0071] This application utilizes TensorRT inference acceleration technology and CUDA parallel computing technology to further improve the inference speed and computational efficiency of the model on edge devices. TensorRT enables more efficient inference on NVIDIA GPUs through graph optimization, operator fusion, and automatic kernel tuning. TensorRT also supports model quantization, which not only reduces model memory footprint but also accelerates the inference process, ensuring stable operation in classroom environments with high real-time requirements. Furthermore, CUDA parallel computing technology is applied to the pre- and post-processing stages. In image preprocessing (such as image scaling and data augmentation) and post-processing (such as Non-Maximum Suppression (NMS)), the parallel computing capabilities of the GPU accelerate these computationally intensive operations, thereby further improving the overall performance of the system.

[0072] In one embodiment of this application, the edge device environment is first configured by installing the Ubuntu 22.04 operating system and the Jetpack development kit (with built-in CUDA and TensorRT) on the Jetson edge device, and configuring the dependency libraries required for model inference (such as OpenCV and TensorRT SDK) to ensure hardware and software environment compatibility. During the model deployment phase, the ONNX model is converted into a TensorRT engine file. During the conversion, graph optimization, operator fusion, and kernel tuning functions are automatically enabled: graph optimization removes redundant computation nodes in the model, simplifying the computation process; operator fusion integrates multiple operations such as convolution + batch normalization + activation into a single operator, reducing memory access latency; and kernel tuning allows TensorRT to automatically generate the optimal computation kernel based on the edge device's GPU architecture, maximizing inference speed. After conversion, the inference performance of the TensorRT engine is tested to ensure that the inference latency of the model on the edge device meets the requirements. For the pre- and post-processing acceleration of the visual model, CUDA parallel processing operators are written to achieve pre- and post-processing acceleration. In the preprocessing stage, an image preprocessing kernel function was written based on CUDA, allocating an independent thread to each pixel. This synchronously performs image scaling (using bilinear interpolation to adjust the image size to the model input size), pixel value normalization (mapping pixel values ​​from 0-255 to the 0-1 range), and BGR-RGB channel conversion. Parallel computation is achieved through CUDA's thread blocks and thread scheduling mechanism, shortening preprocessing time. In the post-processing stage, a prediction box processing kernel function was written, allocating an independent thread to each model's output prediction box. This parallelly performs coordinate decoding (converting the normalized coordinates of the model output to pixel coordinates in the original image) and non-maximum suppression (setting an IOU threshold of 0.5 to remove redundant prediction boxes with high overlap), improving post-processing efficiency. Finally, the CUDA preprocessing and post-processing programs were integrated with the TensorRT inference engine to form a complete edge inference workflow. The processing frame rate of the entire workflow was tested, ensuring an average frame rate exceeding 50fps on edge devices to meet real-time recognition requirements.

[0073] In some embodiments, the method further includes: The results of behavior recognition and analysis are visualized.

[0074] Alternatively, a visualization system can be built using QT, which displays the visual model recognition results in real time and also provides an interactive page with a large model, allowing the analysis of the visual model detection results using a large language model.

[0075] Optionally, the QT visualization system setup requires interface design, database interaction, and function integration to provide teachers with a convenient user interface. The interface design begins with using QT Designer to design three core interfaces: a real-time monitoring interface, a historical data analysis interface, and a report viewing interface. The real-time monitoring interface uses a left-right split layout. The left side uses the QMediaPlayer component to load the classroom video stream transmitted from the edge device, overlaying it with behavior annotation boxes drawn by QPainter. The right side uses the QTableWidget component to display real-time behavior statistics (behavior type, occurrence time, and duration). The historical data analysis interface uses a "filter area - chart area - export area" layout. The filter area uses the QLineEdit and QDateEdit components to provide multi-condition filtering functionality. The chart area uses the QT Charts component to draw line charts, bar charts, and heatmaps. The export area uses the QPushButton component to implement chart export functionality. The report viewing interface uses a left-right split layout. The left side uses the QListWidget component to list all analysis reports, the right side uses the QTextEdit component to display the report content, and export and print buttons are located at the bottom.

[0076] Optionally, in the database interaction function implementation, a local database (SQLite) is connected via QT's QSqlDatabase class, and data read / write interface functions are written: the data storage function inserts the behavior recognition results output by the edge device (including timestamps, class numbers, student IDs, behavior types, etc.) into the database according to a preset format, with the storage frequency consistent with the video frame rate to ensure data integrity; the data query function reads the corresponding historical data from the database based on the filtering conditions input by the teacher (such as class time range, student IDs), and returns structured data for chart display. Simultaneously, a database backup and cleanup mechanism is designed to regularly back up data to prevent loss and automatically clean up invalid data (such as data with recognition confidence below a threshold) to free up storage space.

[0077] Optionally, the visualization system provides teachers with an intuitive interface that displays identified classroom behavior information in real time and presents the analysis results of teaching interactions in a graphical format. Through this module, teachers can instantly view student and teacher behavior patterns, such as whether students are raising their hands or participating in discussions, and whether teachers are writing on the blackboard or lecturing. In addition to real-time display, the system also provides historical data query and analysis functions, allowing teachers to view behavioral data for a specific period to help them better understand student classroom performance and teaching progress.

[0078] Optionally, the visualization system can display the visual model's detection screen in real time, or read historical data from the database to display charts, such as showing the change in the proportion of students exhibiting negative behaviors in the current class compared to the previous class. In addition, the large language model reads information from the database to automatically analyze information such as classroom activity, providing teachers with semantic analysis and helping them optimize classroom teaching.

[0079] Figure 4 This is an overall framework diagram of classroom behavior recognition and analysis provided in the embodiments of this application, such as... Figure 4 As shown in one embodiment of this application, firstly, the visual model detection results serve as the system's data input source. Through processing classroom scene video data, detection information containing various behaviors of students and teachers is output, which forms the basis for subsequent analysis. Next, the database construction phase receives the visual model detection results and stores and manages them according to a preset data structure. The database not only stores real-time detected behavioral data but also provides historical data support for subsequent analysis of large language models, and also provides an interface for data querying and retrieval to the QT terminal page.

[0080] In the deployment and optimization phase of the large language model, a series of techniques, including model quantization, operator fusion, matrix calculation optimization, and system optimization, are used to adapt and accelerate the large language model, enabling it to run efficiently in resource-constrained environments such as edge devices. The optimized large language model reads the required behavioral data from the database and leverages its powerful natural language understanding and analysis capabilities to perform in-depth mining and analysis of classroom behavioral data. Then, the QT terminal page, as the core of the system's interaction and display, receives behavioral data from the database and obtains the analysis results from the large language model.

[0081] Finally, the QT terminal page integrates and processes this information to realize the output of large model analysis results and user interaction, the display of result data, and the display of real-time detection effects of the visual model: The large model analysis result output and user interaction module presents personalized teaching analysis reports generated by the large model to teachers and other users, and supports users to perform interactive operations such as report query and export; The result data display module intuitively displays the statistical data and changing trends of classroom behavior in the form of charts, lists, etc.; The real-time detection effect display module of the visual model presents the detection process and results of classroom behavior by the visual model in real time, such as behavior annotation boxes, behavior types, and other information, allowing users to intuitively understand the real-time dynamics of the classroom.

[0082] The edge computing-based classroom student behavior recognition and analysis system provided in this application is described below. The edge computing-based classroom student behavior recognition and analysis system described below can be referred to in correspondence with the edge computing-based classroom student behavior recognition and analysis method described above.

[0083] Figure 5 This is a schematic diagram of the structure of a classroom student behavior recognition and analysis system based on edge computing, as provided in an embodiment of this application. Figure 5 As shown, the system 500 includes: The optimization module 510 is used to optimize the YOLOv12 model, obtain the optimized YOLOv12 model, and deploy the optimized YOLOv12 model on the edge computing platform. The recognition module 520 is used to input the video data to be recognized and analyzed into the optimized YOLOv12 model to obtain the behavior recognition results. Analysis module 530 is used to analyze the behavior recognition results through a large language model deployed on an edge computing platform to obtain analysis results.

[0084] It should be understood that the above system is used to execute the methods in the above embodiments. The corresponding program modules in the system are similar in implementation principle and technical effect to those described in the above methods. The working process of the system can be referred to the corresponding process in the above methods, and will not be repeated here.

[0085] Based on the methods in the above embodiments, Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown in the illustration, this application provides an electronic device that may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions stored in the memory 630 to execute the edge computing-based classroom student behavior recognition and analysis method described in the above embodiment.

[0086] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the edge computing-based classroom student behavior recognition and analysis method described in the various embodiments of this application.

[0087] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program. When the computer program runs on a processor, it causes the processor to execute the classroom student behavior recognition and analysis method based on edge computing in the above embodiments.

[0088] Based on the methods in the above embodiments, this application provides a computer program product that, when running on a processor, causes the processor to execute the classroom student behavior recognition and analysis method based on edge computing in the above embodiments.

[0089] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0090] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0091] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0092] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0093] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An edge-computing-based classroom student behavior recognition analysis method, characterized in that, The method comprises the following steps: optimizing a YOLOv12 model to obtain an optimized YOLOv12 model, and deploying the optimized YOLOv12 model on an edge computing platform; inputting to-be-identified and analyzed video data into the optimized YOLOv12 model to obtain a behavior recognition result; analyzing the behavior recognition result by a large language model deployed on the edge computing platform to obtain an analysis result. 2.The edge computing based classroom student behavior recognition analysis method of claim 1, wherein, The optimization of the YOLOv12 model comprises the following steps: embedding a two-dimensional deformable attention module in a detection head front feature layer or a feature fusion network of the YOLOv12 model. 3.The edge computing based classroom student behavior recognition analysis method of claim 1, wherein, The optimization of the YOLOv12 model comprises the following steps: replacing a backbone network of the YOLOv12 model with MobileNetV4. 4.The edge computing based classroom student behavior recognition analysis method of claim 1, wherein, The optimization of the YOLOv12 model comprises the following steps: performing network pruning and quantization operations on the YOLOv12 model. 5.The edge computing based classroom student behavior recognition analysis method of claim 1, wherein, The optimization of the YOLOv12 model comprises the following steps: accelerating an inference process of the YOLOv12 model by using TensorRT and CUDA. 6.The edge-computing-based classroom student behavior recognition analysis method according to claim 1, characterized in that, The method further comprises the following steps: visually displaying the behavior recognition result and the analysis result.

7. An edge computing based classroom student behavior recognition analysis system, characterized in that, The method comprises the following steps: an optimization module, configured to optimize a YOLOv12 model to obtain an optimized YOLOv12 model, and deploy the optimized YOLOv12 model on an edge computing platform; an identification module, configured to input to-be-identified and analyzed video data into the optimized YOLOv12 model to obtain a behavior recognition result; an analysis module, configured to analyze the behavior recognition result by a large language model deployed on the edge computing platform to obtain an analysis result.

8. An electronic device, comprising: The computer program product comprises the following steps: at least one memory for storing a computer program; at least one processor for executing the program stored in the memory, and when the program stored in the memory is executed, the processor is configured to execute the edge computing-based classroom student behavior recognition and analysis method according to any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. When the computer program runs on the processor, the processor is caused to execute the edge computing-based classroom student behavior recognition and analysis method according to any one of claims 1-6.

10. A computer program product, characterised in that, When the computer program product runs on the processor, the processor is caused to execute the edge computing-based classroom student behavior recognition and analysis method according to any one of claims 1-6.

Images