Real-time student behavior analysis system for intelligent classroom

Abstract

The invention discloses a student behavior real-time analysis system used in an intelligent classroom. The student behavior real-time analysis system for the intelligent classroom comprises a target detection model and a visualization module. The target detection model comprises an input end, a backbone network, a neck network and a prediction head and is used for processing the initial video image data to obtain a behavior analysis result. And the visualization module is connected to the target detection model and is used for displaying the video image data processed by the target detection model and a final behavior analysis result. The target detection model integrates a bidirectional feature pyramid network and a context enhancement module on the basis of a YOLOv5n detection model; the neck network is established based on a bidirectional feature pyramid network architecture; the context enhancement module is integrated in the backbone network or the neck network.

Description

Technical Field

[0001] The present application relates to the fields of deep learning and computer vision, and specifically to a real-time analysis system for student behavior in a smart classroom. Background Art

[0002] Classroom teaching is an important part of campus management, and student participation in class is crucial to improving overall learning quality and academic progress. The behavior of students in class can directly reflect the quality of the class. Real-time analysis and understanding of students' behavior in class has a positive impact on improving teaching effectiveness. For example, analyzing and evaluating students' behavior in class through deep learning technology can help improve teaching quality and provide feedback.

[0003] At present, there are various conventional evaluation methods for classroom teaching process, such as classroom evaluation scoring system and information-based behavior analysis system. The observation method of students' behavior in the traditional conventional evaluation method of classroom teaching process mainly relies on manual observation. It is difficult for teachers to understand the participation of each student and maintain appropriate interaction. Therefore, it is essential to monitor and analyze students' behavior in real time in the classroom, at least not completely relying on manual observation. However, the camera itself does not have the data processing capability to analyze students' behavior in real time in the classroom.

[0004] Artificial intelligence can be used to expand the function of the camera and realize the autonomous recognition of the behavior or status of students in the classroom. In the field of artificial intelligence, the research of computer vision mainly focuses on the analysis and recognition of video files extracted from imaging devices. The existing behavior detection algorithms for students in the classroom can be roughly divided into three categories: behavior detection based on video action recognition, behavior detection based on posture estimation, and behavior detection based on targets.

[0005] As smart classrooms become more common, analyzing student behavior using classroom cameras combined with deep learning methods has become a prominent research interest in the field of education. Researchers use detection models based on Convolutional Neural Networks (CNN) to directly identify and classify student behavior. For example, one study used an enhanced and faster Region Convolutional Neural Networks (R-CNN) model to identify student behavior in the classroom. This model can directly detect specific behaviors, such as raising hands, standing, and sleeping. Another study introduced a multi-feature-based machine learning method to identify seven types of student behaviors in the classroom. These researchers have conducted extensive research on behavior detection technology and continuously improved detection accuracy, providing a technical framework for behavior detection systems. However, in actual classroom environments, the practical application of these models may face more complex problems. The video analysis technology of some existing behavior detection methods is relatively backward. For example, small target detection, occluded target detection, and real-time detection of educational cameras are difficult to meet actual needs; some behavior detection methods have achieved high accuracy in behavior recognition, but there is still room for improvement in recognition speed.

[0006] In general, existing methods for detecting students’ behavior in the classroom face the following challenges: (1) Real-time, high-precision, and stable detection are challenging. However, in order to solve the challenge of insufficient computing power on edge devices, stable, accurate, and real-time identification and analysis of students’ behavior in the classroom is essential. (2) The occlusion of objects in classroom images (e.g., desks, classmates who are not expected to be detected) leads to fragmentation of visual features, which hinders the accurate identification of occluded objects and classmates who are expected to be detected. (3) The image area of ​​the classmates who are expected to be detected in the classroom images is small and has fewer visual features. The subsequent sampling operation under the deep neural network may have the risk of losing features. (4) The current deep learning model only provides a method for processing video images, but does not present the results of student behavior detection in real time in the classroom. Summary of the invention

[0007] One advantage of the present application is that it provides a real-time analysis system for student behavior in a smart classroom, wherein the real-time analysis system for student behavior in a smart classroom improves the detection speed, recognition accuracy and model generalization ability by specifically designing the target detection model, can realize the detection of target objects with a smaller image area, and improves the detection and recognition ability of occluded objects, and can to a certain extent solve the problem of small-volume target detection objects and occluded objects being difficult to identify in a crowded classroom environment.

[0008] Another advantage of the present application is that it provides a real-time analysis system for student behavior in a smart classroom, wherein the real-time analysis system for student behavior in a smart classroom, through the cooperation of a target detection model and a visualization module, enables the target detection model to present the analysis results of the student behavior in the classroom intuitively and in real time, and can also improve classroom interactivity and fun.

[0009] According to one aspect of the present application, a real-time analysis system for student behavior in a smart classroom is provided, which includes: a target detection model, including an input end, a backbone network, a neck network and a prediction head, for processing initial video image data to obtain a behavior analysis result; a visualization module, connected to the target detection model, for displaying the video image data processed by the target detection model and the final behavior analysis result; wherein the target detection model integrates a bidirectional feature pyramid network and a context enhancement module based on the YOLOv5n detection model; the neck network is established based on the bidirectional feature pyramid network architecture; the context enhancement module is integrated in the backbone network or the neck network.

[0010] In one embodiment of the real-time analysis system for student behavior in a smart classroom according to the present application, the neck network adopts multi-scale dilated convolution; in the multi-scale dilated convolution, the input [bs, C, H, W] is subjected to dilated convolution with convolution rates of 1, 3, and 5, respectively, wherein bs, C, H, and W represent the batch size, number of channels, height, and width of the feature map, respectively.

[0011] In one embodiment of the real-time analysis system for student behavior in a smart classroom according to the present application, the number of convolution kernels of the multi-scale dilated convolution is one-fourth the number of channels input into the neck network.

[0012] In one embodiment of the real-time analysis system for student behavior in a smart classroom according to the present application, the convolution kernel of the multi-scale dilated convolution is a matrix of 3 rows and 3 columns.

[0013] In one embodiment of the real-time analysis system for student behavior in a smart classroom according to the present application, the target detection model is obtained through multiple trainings. During the training process of the target detection model, an optimal transmission allocation strategy is used for label allocation.

[0014] In one implementation of the real-time analysis system for student behavior in a smart classroom according to the present application, the Focal EIoU loss function is used in the optimal transmission allocation strategy.

[0015] In one embodiment of the real-time analysis system for student behavior in a smart classroom according to the present application, a multi-scale training strategy and a sample scaling training strategy are adopted during the training process of the target detection model.

[0016] In one embodiment of the real-time analysis system for student behavior in a smart classroom according to the present application, during the training process of the target detection model, an open source student classroom behavior dataset is used to train the target detection model.

[0017] In one embodiment of the real-time analysis system for student behavior in a smart classroom according to the present application, the visualization module includes a display screen and is configured to display a panoramic image and a close-up image on the display screen.

[0018] In one embodiment of the real-time analysis system for student behavior in a smart classroom according to the present application, the target detection model and the visualization module are configured as follows: the behavior analysis result generated by the target detection model triggers the corresponding display method of the visualization module, and the corresponding display method of the visualization module includes displaying a panoramic image and a close-up image on the same interface.

[0019] Further objectives and advantages of the present application will be fully reflected through understanding of the following description and drawings.

[0020] These and other objects, features and advantages of the present application are fully reflected in the following detailed description, drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or.

[0022] Figure 1 The figure illustrates a schematic diagram of a framework of a real-time analysis system for student behavior in a smart classroom according to an embodiment of the present application.

[0023] Figure 2 The figure illustrates a schematic diagram of a framework of a target detection model for a real-time analysis system of student behavior in a smart classroom according to an embodiment of the present application.

[0024] Figure 3 The figure illustrates a framework diagram of an example of a target detection model for a real-time analysis system of student behavior in a smart classroom according to an embodiment of the present application.

[0025] Figure 4 The figure shows a schematic diagram of the detection effect of the target detection model used in the real-time analysis system of student behavior in the smart classroom according to the embodiment of the present application.

[0026] Figure 5 The figure shows a schematic diagram of an interface of a display screen of a visualization module of a real-time analysis system of student behavior in a smart classroom according to an embodiment of the present application.

[0027] Figure 6 The figure illustrates a schematic diagram of the application process of a real-time analysis system for student behavior in a smart classroom according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0029] It is understood that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple, and the term "one" cannot be understood as a limitation on the number. "Multiple" means greater than or equal to two.

[0030] Although ordinals such as "first," "second," and the like will be used to describe various components, those components are not limited herein. The term is used only to distinguish one component from another. For example, a first component may be referred to as a second component, and likewise, a second component may be referred to as a first component without departing from the teachings of the present application. The term "and / or" as used herein includes any and all combinations of one or more associated listed items.

[0031] The terms used herein are only used for the purpose of describing various embodiments and are not intended to be limiting. As used herein, the singular form is intended to include the plural form, unless the context clearly indicates an exception. It will also be understood that the terms "including" and / or "having" when used in this specification specify the presence of the described features, numbers, operations, components, elements, or combinations thereof, without excluding the presence or addition of one or more other features, numbers, operations, components, elements, or combinations thereof.

[0032] like Figures 1 to 6As shown, a real-time analysis system 100 for student behavior in a smart classroom according to an embodiment of the present application is illustrated. The real-time analysis system 100 for student behavior in a smart classroom improves the detection speed, recognition accuracy and model generalization ability by specifically designing the target detection model 10, can realize the detection of target objects with a small image area, and improves the detection and recognition ability of occluded objects, which can solve the problem of small-volume target detection objects and occluded objects being difficult to recognize in a crowded classroom environment to a certain extent. Figure 1 and Figures 4 to 6 As shown, the real-time analysis system 100 for student behavior in the smart classroom can also present the analysis results of the target detection model 10 on the student behavior in the classroom (for example, raising hands, sleeping, etc.) intuitively and in real time through the cooperation of the target detection model 10 and the visualization module 20, and can also improve the interactivity and fun of the classroom.

[0033] Specifically, in an embodiment of the present application, the existing YOLOv5n detection model is improved to obtain the target detection model 10 of the present application.

[0034] The existing YOLOv5n detection model is one of the versions of the YOLO detection model. YOLO (You Only LookOnce) is a popular computer vision algorithm for real-time object detection. The core idea of ​​YoLo is to divide the entire image into a fixed number of grid cells, and then predict multiple bounding boxes and category probabilities in each grid at the same time.

[0035] The YOLOv5 detection model is one of the most widely used detection models. It is based on deep learning technology and adds modules such as feature pyramid network and spatial pyramid pooling (SPP) structure on the basis of convolutional neural network, thus achieving a balance between high accuracy and fast detection speed.

[0036] The YOLOv5n detection model is a "tiny" ("Nano") version of the YOLOv5 detection model. It is a miniaturized model designed to run on resource-constrained devices. It has a smaller model size and lower computational complexity while maintaining high detection accuracy.

[0037] YOLOv5n mainly includes input terminal 11, backbone network 12 (Backbone), neck network 13 (Neck) and prediction head 14 (Prediction Head). Input terminal 11 is responsible for receiving the original image input, and usually performs some preprocessing operations on the original image, such as adjusting the image size to the input size expected by the model, normalization, etc., to obtain the image processed by input terminal 11. Backbone network 12 is used to extract features from the image processed by input terminal 11. Neck network 13 is used to enhance multi-scale feature fusion and integrate feature maps at different levels. Prediction head 14 is responsible for generating the final prediction output part, which usually contains multiple convolutional layers for predicting the position of the bounding box, category score and object confidence.

[0038] Accordingly, if Figure 1 and Figure 2 As shown, the target detection model 10 of the present application includes an input end 11, a backbone network 12, a neck network 13 and a prediction head 14. The input end 11 of the target detection model 10 of the present application is formed based on the input end 11 of the YOLOv5n detection model or is consistent with the input end 11 of the YOLOv5n detection model, and has data enhancement, adaptive anchor frame calculation, and adaptive image scaling functions. The prediction head 14 of the target detection model 10 of the present application is formed based on the prediction head 14 of the YOLOv5n detection model or is consistent with the prediction head 14 of the YOLOv5n detection model.

[0039] The backbone network 12 of the target detection model 10 of the present application is mainly responsible for extracting features from the image input therein using convolution downsampling technology, which helps to extract the key features of the image, wherein these key features are essential for accurate analysis and classification. The backbone network 12 of the YOLOv5n detection model includes multiple convolutional layers, including a slice (Focus) structure, a convolution normalization (Convolution-Batch Normalization-Leaky ReLU, CBL) structure, a cross-stage partial (Cross Stage Partial, CSP) structure and a fast spatial pyramid pooling (Spatial PyramidPooling fast, SPPF) structure. In one embodiment of the present application, the backbone network 12 of the target detection model 10 at least retains the parameters of the fifth convolutional layer, the seventh convolutional layer and the ninth convolutional layer of the backbone network 12 of the YOLOv5n detection model, and prepares for the feature fusion of the neck network 13.

[0040] In an embodiment of the present application, a bidirectional feature pyramid network 131 (Bidirectional Feature Pyramid Network, BIFPN) and a context augmentation module 132 (Context augmentation Module, CAM) are integrated into the YOLOv5n detection model to improve detection speed and recognition accuracy.

[0041] The bidirectional feature pyramid network 131 is formed on the basis of the feature pyramid network (BIFPN), and can realize bidirectional information flow, that is, there is not only a path from high-resolution low-level feature maps to low-resolution high-level feature maps, but also a path from low-resolution high-level feature maps to high-resolution low-level feature maps, which can more effectively fuse multi-scale feature information.

[0042] The basic principle of the context enhancement module 132 is to obtain the importance score of each input feature by re-weighting the weight of the last convolutional layer. Specifically, the convolutional layer output is firstly globally average pooled, and then the obtained vector is element-wise multiplied and summed with the weight vector of the convolutional layer to finally obtain the score of each feature.

[0043] Specifically, in the embodiments of the present application, Figure 3 As shown, the neck network 13 is established based on the bidirectional feature pyramid network 131 architecture, so that the shallow features have accurate positioning information, which is conducive to the recognition of small-volume target detection objects or target detection objects with a small image area; it also makes the deep features have rich semantic information, which is conducive to the recognition and classification of large-volume target detection objects or target detection objects with a large image area. The neck network 13 formed based on the bidirectional feature pyramid network 131 architecture extracts features from multiple convolutional layers of the backbone network 12 to generate a feature pyramid, and adopts a weighted feature fusion mechanism to combine features of different levels. The fusion weights are learned during the training process.

[0044] In an embodiment of the present application, the context enhancement module 132 is integrated into the backbone network 12 or the neck network 13 established based on the bidirectional feature pyramid network 131 architecture. In one embodiment of the present application, the context enhancement module 132 is integrated into the backbone network 12, specifically, the last layer of the SPPF structure of the backbone network 12 of the YOLOv5n detection model is replaced by the CAM module. In another embodiment of the present application, the context enhancement module 132 is integrated into the neck network 13 established based on the bidirectional feature pyramid network 131 architecture, such as Figure 3As shown, the context information of small-volume target detection objects or target detection objects with small image areas is improved.

[0045] The present application combines the context enhancement module 132 and the bidirectional feature pyramid network 131 to form a feature pyramid composite structure, which can use the feature fusion of multi-scale void convolution to supplement the context information in the network; in addition, it can also use the channel and space feature extraction mechanism to suppress the conflicting information after multi-scale feature fusion, thereby preventing small targets from being submerged in the conflicting information.

[0046] In order to solve the problem that the behavior detection of students who are far away from the camera in the classroom has fuzzy texture and contains few features, which makes it difficult to detect, the neck network 13 established based on the bidirectional feature pyramid network 131 architecture that integrates the context enhancement module 132 adopts multi-scale dilated convolution and feature fusion to enhance the network's ability to capture features of different scales. In the multi-scale dilated convolution, the input [bs, C, H, W] is subjected to dilated convolution with convolution rates of 1, 3, and 5, respectively, where bs, C, H, and W represent the batch size, number of channels, height, and width of the feature map, respectively. Considering the input of the small-sized image of the target detection model 10, in order to obtain more detailed features, the convolution kernel in the multi-scale hole convolution is 3×3, that is, a matrix of 3 rows and 3 columns; the number of convolution kernels is one-fourth of the number of channels input to the neck network 13 to reduce the amount of parameters introduced, that is, first compressing the number of channels to one-fourth of the number of channels input to the neck network 13, which helps to reduce the computational cost, and then expanding the number of channels compressed to one-fourth of the number of channels input to the neck network 13 to the number of channels input to the neck network 13 through a convolution with a convolution kernel of 1×1 to ensure the consistency of subsequent fusion operations, thereby obtaining outputs of three different receptive fields; finally, the features are fused. The neck network 13 established based on the bidirectional feature pyramid network 131 architecture with the context enhancement module 132 integrated also uses the channel and space feature extraction mechanism to suppress the conflicting information after multi-scale feature fusion, thereby preventing small targets from being submerged in the conflicting information.

[0047] The target detection model 10 is obtained after multiple trainings. Selecting high-quality training samples is the key to obtaining accurate behavior analysis results, among which assigning positive or negative labels to samples is challenging. Especially when dealing with overlapping targets, inappropriate label assignment may lead to misclassification of positive and negative samples, thereby reducing the accuracy of the network.

[0048] It is worth mentioning that in order to solve the problem of ambiguous label assignment, in the embodiment of the present application, the optimal transport assignment (OTA) strategy is adopted during the training process of the target detection model 10. The optimal transport assignment strategy examines the label assignment from a global perspective, and proposes to express the assignment procedure as an optimization transportation problem, defining the unit transportation cost between each anchor box and target pair as the weighted sum of its classification and regression losses. In order to obtain the minimum overall loss and find the best transportation plan, the optimal transport assignment strategy converts the search for the best allocation plan into the best transportation plan that minimizes the transportation cost.

[0049] In the optimal transmission allocation strategy, each element in the transportation cost matrix of the optimal transmission allocation strategy, that is, the "transportation cost" between the anchor box and the real target, is defined by a loss function to improve the detection of occluded objects, so that the convergence speed is faster and the positioning results are better. In the target detection task, accurately defining the loss function can improve the performance of the target detection model 10. The commonly used IoU-based loss function may not be the best choice for classroom behavior recognition tasks. The CIoU metric shows higher sensitivity to small-volume target detection objects or target detection objects with smaller image areas. This sensitivity may lead to misclassification of targets, thereby causing a decrease in model performance; in order to address these limitations, in an embodiment of the present application, the Focal EIoU (Focal Enhanced Intersection over Union) loss function is used in the optimal transmission allocation strategy. The Focal EIoU loss function is formed by combining Focal Loss and EIoU Loss. Focal Loss splits the width and height of the prediction box and performs difference operations with the width and height of the minimum external box respectively. EIoULoss reduces the difference in width and height between the predicted box and the real box, making the convergence faster and achieving better positioning results. The Focal EIoU loss function is designed to address the shortcomings of traditional loss functions in dealing with overlapping targets, unbalanced category distribution, and positioning accuracy.

[0050] It is also worth mentioning that in the training process of the target detection model 10, a multi-scale training strategy and a sample scaling training strategy are also adopted to improve the detection and recognition capabilities of small-volume target detection objects or target detection objects with a small image area, and enhance the robustness of the target detection model 10. The multi-scale training strategy refers to the use of images of different resolutions as input during the training process, so that the model can learn target features of different scales. The core idea is to enhance the model's adaptability to targets of different scales through diversified input images. Sample scaling training refers to scaling the input image and changing its resolution during the training process to increase the diversity of the data. The core idea is to enable the model to be trained at different scales through scaling operations, thereby enhancing its adaptability to scale changes.

[0051] During the training process of the target detection model 10, images of different scales are input, and a specific scale is randomly selected in each training iteration. This helps to improve the stability of the target detection model 10 and accept images of different sizes.

[0052] When processing dense targets in the embodiment of the present application, the multi-scale training strategy and the sample scaling strategy can help the target detection model 10 better capture targets of different sizes and avoid missed detections and misclassifications. For small target detection tasks, the multi-scale training strategy provides more detailed information, while the sample scaling strategy ensures that the model can effectively detect these targets and improve detection accuracy. In complex scenarios, the multi-scale training strategy and the sample scaling strategy can help the target detection model 10 cope with changing environments and improve overall performance.

[0053] It is also worth mentioning that during the training process of the target detection model 10, an open source Student Classroom Behavior (SCB) dataset is used to train the target detection model 10.

[0054] In summary, in the embodiment of the present application, the bidirectional feature pyramid network 131 and the context enhancement module 132 are combined to form a feature pyramid composite structure to prevent small targets from being submerged in conflicting information, and the OTA strategy is introduced in the training process of the target detection model 10 to solve the problem of difficult identification of occluded objects, and the stability of the target detection model 10 and its adaptability to changing and complex environments are improved through multi-scale training strategies and sample scaling strategies. The feature pyramid composite structure, OTA strategy, multi-scale training strategy and sample scaling strategy can significantly enhance the detection and recognition capabilities of the target detection model 10 for smaller targets and occluded targets, solve the problem of difficult identification of small objects and occluded objects in crowded classroom environments to a certain extent, and improve the model generalization capability of the target detection model 10.

[0055] In order to verify the performance of the target detection model 10, a performance test was performed on the target detection model 10. Specifically, 450 frames of images extracted from classroom videos were analyzed by the target detection model 10. The experimental results show that the target detection model 10 can solve two common challenges in classroom behavior recognition: detecting small-volume target detection objects or target detection objects with small image areas and identifying occluded objects. Specifically, in a classroom environment, due to the existence of occlusion, traditional detection models may not be able to recognize the behavior of occluded students. The target detection model 10 of the present application can recognize the behavior of occluded objects and occluded objects, showing strong robustness and stability, and low computational cost. It can maintain a high recognition accuracy and recognition speed in real classroom scenes, with an average recognition accuracy of 82.6%, and an average recognition speed of 20.6 frames per second on the front-end device, which exceeds the lightweight target detector. The target detection model 10 of the present application is very suitable for classroom behavior detection tasks, and is suitable for being deployed on embedded platforms such as classroom cameras, and applied to real-time classroom behavior recognition. It can accurately and efficiently analyze and provide feedback on the behavior of students in the classroom based on educational cameras with insufficient computing power.

[0056] As mentioned above, the real-time analysis system 100 for student behavior in the smart classroom also enables the target detection model 10 to present the analysis results of the behavior of students in the classroom intuitively and in real time through the cooperation of the target detection model 10 and the visualization module 20, and can also improve the interactivity and fun of the classroom. The visualization module 20 is connected to the target detection model 10 and includes at least one display screen. The display screen is used to display the final video image data and the final behavior analysis results. The final video image data and the final behavior analysis results can be displayed on the global wide area network (Web) application through the HyperText Transfer Protocol (HTTP) protocol. Accordingly, the behavior of students in the classroom can be mapped to the global wide area network (Web) application in the form of video, so that the teacher can intuitively see the final behavior analysis results of each student in the classroom in real time.

[0057] In one embodiment of the present application, the visualization module 20 can visualize the details of students' behaviors in the classroom to improve the classroom teaching effect; the visualization module 20 can also trigger the corresponding display method of the visualization module 20 through the behavior analysis results generated by the target detection model 10, thereby enhancing the real-time interactive communication between teachers and students in the classroom and increasing the fun and interactivity of teaching.

[0058] In one example of the present application, a user can issue instructions such as lens stretching, rotation, and detection through the visualization module 20 or remote control, so that the visualization module 20 displays corresponding images based on the instructions such as lens stretching, rotation, and detection, for example, displaying panoramic images and close-up images on the display screen of the visualization module 20.

[0059] In another example of the present application, when the behavior analysis result generated by the target detection model 10 is raising hands, sleeping, or other specific behavior analysis results, the local close-up mode of the visualization module 20 is triggered, and the panoramic image and the close-up image (such as Figure 1 and Figure 5 As shown in the figure, the image of students in the classroom can be enlarged to show the details of their behavior in the classroom. The panoramic image can be displayed on the left side of the display screen, and the close-up image can be displayed on the right side of the display screen. If there are multiple people interacting, the multiple people will be selected by stretching the lens. By enlarging the close-up, a clearer display can be provided, which increases the fun and interactivity of teaching and improves students' interest in learning.

[0060] In one embodiment of the present application, the real-time analysis system 100 for student behavior in a smart classroom further includes a controller and an intelligent display chip 40. The controller is connected to the target detection model 10, and is used to receive the video image data and behavior analysis results processed by the target detection model 10. The intelligent display chip 40 is connected to the visualization module 20 and the controller, and is used to realize data transmission between the visualization module 20 and the controller.

[0061] like Figure 1 and Figure 6As shown, in an actual classroom, the initial video image data of the actual classroom can be first collected through a camera; then the target detection model 10 is used to perform data preprocessing, feature extraction, feature fusion, and behavior analysis result prediction on the initial video image data; then the video image data and behavior analysis results processed by the target detection model 10 are processed by the processor 30, wherein the processor 30 can be selected according to needs, for example, Cortex-A5; then, the video image data and behavior analysis results processed by the processor 30 are transmitted to the visualization module 20 through the intelligent display chip 40 (for example, SSD261Q main control board), or the video image data and behavior analysis results processed by the target detection model 10 are transmitted to the visualization module 20, so that the final video image data (that is, the predicted video image data processed by the processor 30 or the video image data processed by the target detection model 10) and the final behavior analysis results (that is, the behavior analysis results processed by the processor 30 or the behavior analysis results processed by the target detection model 10) are displayed on the display screen of the visualization module 20. The intelligent display chip 40 and the visualization module 20 can perform data transmission via the hypertext transfer protocol. The intelligent display chip 40 can receive instructions such as lens stretching, rotation and detection from the visualization module 20.

[0062] In summary, the real-time analysis system 100 for student behavior in a smart classroom according to the embodiment of the present application is explained. The real-time analysis system 100 for student behavior in a smart classroom improves the detection speed, recognition accuracy and model generalization ability by specifically designing the target detection model 10, can realize the detection of target objects with a small image area, and improves the detection and recognition ability of occluded objects, which can solve the problem of small-volume target detection objects and occluded objects being difficult to identify in a crowded classroom environment to a certain extent; through the cooperation of the target detection model 10 and the visualization module 20, the analysis results of the target detection model 10 on the behavior of students in the classroom can be presented intuitively and in real time, and the classroom interactivity and fun can also be improved.

[0063] The present application and its implementation methods are described above, and such description is not restrictive. The drawings show only one implementation method of the present application, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it and design structural methods and embodiments similar to the technical solution without creative design without departing from the inventive purpose of the present application, they should all fall within the protection scope of the present application.

Claims

1. A real-time analysis system for student behavior in a smart classroom, characterized in that: include: The target detection model, including the input end, backbone network, neck network and prediction head, is used to process the initial video image data to obtain the behavior analysis results; A visualization module is connected to the target detection model and is used to display the video image data processed by the target detection model and the final behavior analysis results; wherein the target detection model integrates a bidirectional feature pyramid network and a context enhancement module based on the YOLOv5n detection model; the neck network is established based on the bidirectional feature pyramid network architecture; the context enhancement module is integrated into the backbone network or the neck network.

2. The real-time analysis system for student behavior in a smart classroom according to claim 1 is characterized in that: The neck network adopts multi-scale dilated convolution; in the multi-scale dilated convolution, dilated convolutions with convolution rates of 1, 3, and 5 are performed on the input [bs, C, H, W], respectively, where bs, C, H, and W represent the batch size, number of channels, height, and width of the feature map, respectively.

3. The real-time analysis system for student behavior in a smart classroom according to claim 2 is characterized in that: The number of convolution kernels of the multi-scale dilated convolution is one quarter of the number of channels input into the neck network.

4. The real-time analysis system for student behavior in a smart classroom according to claim 2 is characterized in that: The convolution kernel of the multi-scale dilated convolution is a matrix with 3 rows and 3 columns.

5. The real-time analysis system for student behavior in a smart classroom according to claim 1 is characterized in that: The target detection model is obtained through multiple trainings. During the training process of the target detection model, an optimal transmission allocation strategy is used for label allocation.

6. The real-time analysis system for student behavior in a smart classroom according to claim 5 is characterized in that: The Focal EIoU loss function is adopted in the optimal transmission allocation strategy.

7. The real-time analysis system for student behavior in a smart classroom according to claim 6 is characterized in that: During the training process of the target detection model, a multi-scale training strategy and a sample scaling training strategy are adopted.

8. The real-time analysis system for student behavior in a smart classroom according to claim 7 is characterized in that: During the training process of the target detection model, an open source student classroom behavior dataset is used to train the target detection model.

9. The real-time analysis system for student behavior in a smart classroom according to claim 1 is characterized in that: The visualization module includes a display screen and is configured to display a panoramic image and a close-up image on the display screen.

10. The real-time analysis system for student behavior in a smart classroom according to claim 9, characterized in that: The target detection model and the visualization module are configured as follows: the behavior analysis result generated by the target detection model triggers the corresponding display method of the visualization module, and the corresponding display method of the visualization module includes displaying a panoramic image and a close-up image on the same interface.

Images