In-class student behavior detection and identification method based on deep learning

Abstract

The invention relates to a method for detecting and recognizing behaviors of students behind a classroom based on deep learning. The method comprises the following steps: firstly, acquiring behavior image data of students behind a classroom; constructing a behavior detection and recognition model based on the YOLOX-m target detection network, and optimizing a trunk feature extraction module, a neck feature enhancement module and a prediction head module of the YOLOX-m target detection network; training the behavior detection and recognition model based on the behavior image data of the students behind the classroom; and finally, inputting a to-be-recognized actual classroom image into the trained behavior detection and recognition model, and outputting a behavior detection and recognition result. According to the method, the head raising behavior and the head lowering behavior of the student in class can be accurately detected and recognized from the back by collecting images from the back of the class and optimizing all modules of an existing target detection network and combining an attention mechanism and deep separable convolution, and the learning state of the student can be timely and effectively reflected by detecting and recognizing the two typical behaviors.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a method for detecting and identifying student behavior at the back of a classroom based on deep learning. Background Art

[0002] As a new form of education in the digital age, smart education effectively promotes the innovation of the education system. The classroom is the main scene of smart education and the core and main battlefield of the digital transformation of school education. In classroom teaching activities, students are the main body of learning activities, and their behavior status is a direct reflection of teaching activities. The analysis of students' classroom behavior is not only an important part of teaching analysis, but also an important factor affecting students' learning and teachers' teaching efficiency. With the development of deep learning technology, education has gradually entered the era of intelligence. How to achieve modern management of classroom teaching has become the focus of discussion in the field of education, and in-depth analysis of classroom teaching behavior has become possible.

[0003] In traditional classrooms, teachers obtain feedback on teaching by observing students' performance. However, this one-to-many method has problems such as incomplete observation, low credibility, poor real-time performance, and time-consuming and labor-intensive. The method of identifying students' behavioral status in the classroom based on deep learning target detection algorithm breaks the traditional classroom teaching activity analysis method. It uses various types of convolutional networks to classify and identify students' classroom behavior actions, reflect students' class situation in real time, and intuitively and dynamically quantify students' classroom status, helping teachers to fully grasp students' classroom status, timely and reasonably adjust teaching progress and strategies, and improve teaching efficiency.

[0004] At present, deep learning technology in classroom behavior analysis research mainly focuses on behavioral action research, evaluating students' behavior in the classroom through their behavioral postures, and combining target detection technology to reasonably analyze problems in educational scenarios and solve them in a targeted manner.

[0005] Previous researchers have conducted some methodological research in the field of smart classrooms and student classroom behavior detection. However, there are still the following deficiencies: (1) Most existing methods conduct relevant experimental research on a single student or a specially built virtual classroom, and no relevant experiments are conducted in a real multi-person large classroom; (2) The datasets used in the existing methods are all shot from the front to the back of the classroom, exposing the faces of most students, infringing on the students' portrait rights, and have not considered the impact of the Hawthorne effect on the experimental data; at the same time, the existing datasets have many problems such as occlusion, angle differences, and differences in student ratios. (3) The detection model used by the existing methods is not accurate enough and no experiments have been conducted on the detection speed. In view of the actual situation, the use scenario of the behavior detection model is mostly deployed in deep learning edge devices for use, requiring the detection model to have high detection accuracy while ensuring the lightweight of the detection model. The above studies mostly use the benchmark target detection model for experiments, and have not optimized the detection model.

[0006] Therefore, in the relevant technology, there is an urgent need for a method that can protect the privacy of students' portraits and ensure the effectiveness of classroom behavior detection and recognition. Summary of the invention

[0007] Based on this, it is necessary to provide a method for detecting and identifying student behavior in the back of the classroom based on deep learning, which can protect the privacy of students' portraits and ensure the effect of classroom behavior detection and identification, in response to the above technical problems.

[0008] In a first aspect, the present application provides a method for detecting and identifying student behavior at the back of a classroom based on deep learning. The method comprises:

[0009] Obtain image data of student behavior at the back of the classroom;

[0010] Build a behavior detection and recognition model based on the YOLOX-m target detection network, and optimize the backbone feature extraction module, neck feature enhancement module and prediction head module of the YOLOX-m target detection network;

[0011] Training the behavior detection and recognition model based on the behavior image data of students in the back of the classroom;

[0012] The actual classroom image to be identified is input into the trained behavior detection and recognition model, and the behavior detection and recognition result is output.

[0013] Optionally, in one embodiment of the present application, obtaining the student behavior image data at the back of the classroom includes:

[0014] Mark students' behavior in class.

[0015] Optionally, in one embodiment of the present application, the optimized backbone feature extraction module includes:

[0016] A module based on the MSCA attention mechanism is added to each output layer of the backbone feature extraction module to improve the feature extraction capability of the behavior detection and recognition model.

[0017] Optionally, in one embodiment of the present application, the neck feature optimization and enhancement module includes:

[0018] A depth-wise separable convolution minimum module DBS is constructed to reduce the number of parameters and the amount of calculation of the behavior detection and recognition model.

[0019] Optionally, in one embodiment of the present application, the optimization prediction head module includes:

[0020] The GIoU loss function is used instead of the IoU loss function to speed up the model fitting the position relationship between the predicted box and the real box.

[0021] Optionally, in one embodiment of the present application, the method further includes:

[0022] The detection capability of the behavior detection and recognition model is evaluated using precision, recall, detection accuracy and average detection accuracy.

[0023] In a second aspect, the present application also provides a device for detecting and identifying student behavior at the back of a classroom based on deep learning. The device comprises:

[0024] A data acquisition module is used to acquire the image data of student behavior at the back of the classroom;

[0025] A model building module is used to build a behavior detection and recognition model based on the YOLOX-m target detection network, and optimize the backbone feature extraction module, neck feature enhancement module and prediction head module of the YOLOX-m target detection network;

[0026] A model training module, used for training the behavior detection and recognition model based on the behavior image data of students in the back of the classroom;

[0027] The behavior detection and recognition module is used to input the actual classroom image to be recognized into the trained behavior detection and recognition model and output the behavior detection and recognition result.

[0028] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor executes the steps of the methods described in the above embodiments.

[0029] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in each of the above embodiments are implemented.

[0030] The above-mentioned method for detecting and identifying student behaviors at the back of the classroom based on deep learning first obtains the student behavior image data at the back of the classroom; then, a behavior detection and identification model is constructed based on the YOLOX-m target detection network, and the backbone feature extraction module, neck feature enhancement module and prediction head module of the YOLOX-m target detection network are optimized; then, the behavior detection and identification model is trained based on the student behavior image data at the back of the classroom; finally, the actual classroom image to be identified is input into the trained behavior detection and identification model, and the behavior detection and identification result is output. In other words, by collecting images from the back of the classroom, optimizing the various modules of the existing target detection network, and combining the attention mechanism and the deep separable convolution, it is possible to accurately detect and identify the two behaviors of students raising their heads and lowering their heads in class from the back. By detecting and identifying these two typical behaviors, it is possible to timely and effectively reflect the students' learning status, thereby helping teachers to accurately grasp the students' learning situation, better improve the classroom, thereby improving the efficiency of teaching and learning, and further facilitating intelligent classroom teaching. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A diagram of an application environment of a method for detecting and identifying student behavior at the back of a classroom based on deep learning in one embodiment;

[0032] Figure 2 A flowchart of a method for detecting and identifying student behavior at the back of a classroom based on deep learning in one embodiment;

[0033] Figure 3 A schematic diagram of the structure of a behavior detection and recognition model in one embodiment;

[0034] Figure 4 A schematic diagram of the structure of an optimized backbone feature extraction module in one embodiment;

[0035] Figure 5 A schematic diagram of the structure of the MSCA attention mechanism in one embodiment;

[0036] Figure 6 Schematic diagram of the structure of depthwise separable convolution in one embodiment;

[0037] Figure 7 Schematic diagram of a GIoU loss function in one embodiment;

[0038] Figure 8 A schematic diagram of a behavior detection and recognition result of a real classroom scene in an embodiment;

[0039] Fig. 9 It is a structural block diagram of a device for detecting and identifying student behavior at the back of a classroom based on deep learning in one embodiment;

[0040] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0042] The present application embodiment provides a method for detecting and identifying student behavior at the back of a classroom based on deep learning, which can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0043] In one embodiment, Figure 2 As shown in the figure, a method for detecting and identifying student behavior at the back of the classroom based on deep learning is provided. Figure 1 The server in the example is used to illustrate the following steps:

[0044] S201: Obtaining student behavior image data at the back of the classroom.

[0045] In the embodiment of the present application, first, in order to ensure that the research data can truly and objectively reflect the classroom behavior of students, the real classroom scene of a certain university is collected. Specifically, the EZVIZ C6c surveillance camera is selected for shooting, and this camera uses a 5-megapixel ultra-clear lens. The surveillance camera is set up at the back of the classroom, and the appropriate angle is adjusted to capture the video of the real teaching scene. A total of 10.75 hours of effective video time is collected. The collected classroom video is cut and the key frames are intercepted, and a total of 962 valid data are obtained. The valid data is divided and pre-processed to obtain a data set for training.

[0046] Specifically, in one embodiment of the present application, obtaining the student behavior image data at the back of the classroom includes:

[0047] Mark students' behavior in class.

[0048] In one embodiment of the present application, LabelImg software is used to label students' behaviors in the classroom. In pedagogy, the indicators related to the measurement of learning behavior engagement can be explored from the two dimensions of engagement and disengagement, wherein sitting upright and looking forward are regarded as engaged behaviors, while lowering the head and overlapping the head with the desktop are regarded as disengaged behaviors. Therefore, in the classification and division of the data set, combined with the relevant theories in pedagogy, the student behaviors in college classrooms are macroscopically divided into two states: listening to the class and being distracted. Students who raise their heads are in the state of listening to the class, while those who lower their heads are in the state of being distracted. In the specific application, students who raise their heads to listen to the class are marked as one category, and students who lower their heads and are distracted are marked as another category, and a total of 51,955 detection frames are obtained. Referring to the definition of target size in the COCO dataset in target detection, targets with a total number of pixels less than 32×32 in area are defined as small targets; targets with a total number of pixels in the range of 32×32 to 96×96 are defined as medium-sized targets; targets with a total number of pixels greater than 96×96 are defined as large-sized targets. There are 32084 small targets, 19833 medium-sized targets, and 38 large targets in the dataset. Therefore, classroom student behavior detection can be regarded as a detection task for small and medium-sized targets.

[0049] S203: Building a behavior detection and recognition model based on the YOLOX-m target detection network, and optimizing the backbone feature extraction module, neck feature enhancement module and prediction head module of the YOLOX-m target detection network.

[0050] In the embodiment of the present application, YOLOX is a new type of single-stage target detection network. Compared with the YOLO series network, its improvements mainly include data enhancement, prediction branch decoupling and no anchor points. YOLOX will effectively fuse the features of three different levels and then decode them, so it has a better detection effect for small targets. In the YOLOX series model, YOLOX-m has a small number of parameters, which is convenient for deployment on edge devices and can provide support for the visual system of deep learning embedded devices. Therefore, the YOLOX-m target detection network is selected as the baseline model, and optimization and improvement are carried out on this basis. The optimized parts mainly include the trunk feature extraction module, the neck feature enhancement module and the prediction head module, such as Figure 3 As shown, it is a structural diagram of the behavior detection and recognition model.

[0051] Specifically, in one embodiment of the present application, the optimized backbone feature extraction module includes:

[0052] A module based on the MSCA attention mechanism is added to each output layer of the backbone feature extraction module to improve the feature extraction capability of the behavior detection and recognition model.

[0053] In one embodiment of the present application, in a real classroom scene, since the upper half of the students are in an active state, each student is easily blocked by adjacent students. In the original YOLOX-m, CSPDarkNet is used as the backbone extraction network to mainly extract the input image, and then the image is divided into cells for detection. The network needs to process each part of the image. These operations have defects such as large amount of calculation, low calculation efficiency, and inability to accurately detect each student, resulting in problems such as missed detection and false detection. Therefore, Figure 4 As shown in the figure, a module based on the MSCA (Multi-Scale Convolutional Attention) attention mechanism is added to each output layer of the backbone feature extraction module to improve the feature extraction capability of the behavior detection and recognition model on student targets. Figure 5 The figure shows the structure of the module based on the MSCA attention mechanism. The design of the attention mechanism mainly includes three parts: deep convolution for aggregating local information, multi-branch deep strip convolution for capturing multi-scale context, and a 1×1 convolution as the weight of the entire attention. In each branch, two deep strip convolutions are used to simulate the standard deep convolution with a large convolution kernel. The calculation formula is as follows:

[0054]

[0055] Among them, F represents the input feature, Att and Out refer to the attention map and output respectively. The symbol is used to represent the element-wise matrix multiplication operation. DW-Conv refers to depthwise convolution, while Scale i (i∈{0,1,2,3}) refers to the i-th branch in the figure. Scale0 is an identity connection. In each branch, the MSCA attention mechanism uses two deep striped convolutions to approximate the standard deep convolution with a large kernel, and sets the convolution kernel size of each branch to 7, 11, and 21 respectively.

[0056] In this embodiment, the deep stripe convolution of the MSCA attention mechanism can more effectively extract the characteristics of the student target. At the same time, the larger convolution kernel can combine local information and global information, enhance the understanding of the complex scene of the actual classroom, and enhance the meticulousness and accuracy of the detection model. By adding modules based on the MSCA attention mechanism to the output layers of different scales, more effective image features and semantic information can be extracted, greatly improving the accuracy of the entire detection model.

[0057] Specifically, in one embodiment of the present application, the neck feature optimization and enhancement module includes:

[0058] A depth-wise separable convolution minimum module DBS is constructed to reduce the number of parameters and the amount of calculation of the behavior detection and recognition model.

[0059] In one embodiment of the present application, in YOLOX, the original neck feature enhancement module FPN mainly uses the CBS minimum module for enhanced feature extraction. However, the CBS module uses ordinary standard convolution, resulting in a large amount of calculation and parameter. Therefore, a deep separable convolution minimum module DBS is constructed, and a deep separable convolution with a smaller amount of calculation and parameter is used to replace the ordinary convolution, so as to reduce the parameter amount and calculation amount of the entire behavior detection and recognition model.

[0060] like Figure 6 As shown in the figure, the depthwise convolution consists of two parts, namely the spatial convolution (DW) and the channel convolution (PW). First, the depthwise separable convolution processes the spatial information in the length and width directions, divides the input image into a single layer of channels, and uses a two-dimensional convolution kernel to calculate the single channel convolution, which is equivalent to the vector dot product of two matrices. Each channel has a dedicated convolution kernel, and the number of input channels is used to generate the corresponding feature map. Then the channel information in the stacked feature map is convolved using the unit convolution kernel to supplement the lost cross-channel information, and finally a new feature map is obtained.

[0061] In this embodiment, using depthwise separable convolution can reduce the number of parameters and the amount of computation and improve the efficiency of convolution operations.

[0062] Specifically, in one embodiment of the present application, the optimization prediction head module includes:

[0063] The GIoU loss function is used instead of the IoU loss function to speed up the model fitting the position relationship between the predicted box and the real box.

[0064] In one embodiment of the present application, in the classroom student behavior detection task, the original YOLOX model uses IoU (Intersection over Union) to characterize the degree of overlap between the real box and the predicted box, but cannot explain the distance between the real box and the predicted box. If there is no intersection between the predicted box and the real box, the calculation result of IoU will always be 0. At this time, no matter how far or near the distance between the two boxes is, IoU cannot be reflected; and IoU cannot distinguish the overlapping form between the two targets. When one of the two boxes is inside the other, if the size of the box remains unchanged, regardless of the alignment method, the IoU value will not change. Therefore, the GIoU (generalized intersection over union) loss function is used to replace the original evaluation index IoU loss function and is introduced into YOLO-Head to speed up the model fitting the positional relationship between the predicted box and the real box, so as to improve the accuracy of the student detection regression box. The GIoU loss function can ensure that the regression of the target detection box is scale invariant, while keeping the optimization target and loss function of the detection box consistent. In GIoU, the distance between the two boxes is no longer determined solely by the information of the predicted box and the actual box. As Figure 7 As shown in the figure, by introducing the minimum circumscribed rectangle Ac and adding the information of D, the distance information between the two boxes is marked by A, B, Ac and D. Even if A and B do not overlap, GIoU can reflect the distance between the two boxes. If the two targets do not completely overlap, their overlapping method will show the degree of alignment and similarity. The closer the alignment between the two targets, the higher the similarity of the targets. By introducing Ac and D, GIoU can better reflect the degree of alignment of the detection box, and the evaluation is more accurate, thereby improving the detection performance of the algorithm.

[0065] S205: Training the behavior detection and recognition model based on the student behavior image data at the back of the classroom.

[0066] In the embodiment of the present application, the student behavior image data at the back of the classroom is divided into a training set, a validation set, and a test set in a ratio of 8:1:1 to prevent the detection model from overfitting. The input image is resized to 608×608, and each model is trained for 150 rounds. The backbone network is frozen and trained using the idea of ​​transfer learning. The first 50 rounds are frozen, and the initial learning rate is set to 0.001, the batch size is set to 8, and the weight decay is 0.0005; the next 100 rounds are unfrozen, and the initial learning rate is set to 0.0001, the batch size is set to 4, and the weight decay is 0.0005.

[0067] In one embodiment of the present application, the method further includes:

[0068] The detection capability of the behavior detection and recognition model is evaluated using precision, recall, detection accuracy and average detection accuracy.

[0069] In one embodiment of the present application, precision, recall, average precision (AP) and mean average precision (mAP) are used to evaluate the detection capability of the trained behavior detection and recognition model. The specific formula is as follows:

[0070]

[0071] Among them, TP means that positive samples are predicted as positive samples (True Positives), FP means that negative samples are predicted as positive samples (False Positives), FN means that positive samples are predicted as negative samples (False Negatives), P means precision, that is, the proportion of samples predicted as positive that are actually positive samples, and R means recall, that is, the proportion of all positive samples that are actually predicted as positive samples.

[0072] S207: Input the actual classroom image to be identified into the trained behavior detection and recognition model, and output the behavior detection and recognition result.

[0073] In the embodiment of the present application, actual classroom pictures are randomly selected from different classrooms, and after the pictures are resized to 608×608 pixels, they are input into the trained behavior detection and recognition model for prediction, and the behavior detection and recognition results are output. Figure 8 The figure shows a schematic diagram of the behavior detection and recognition results in a real classroom scene. The red prediction box represents the students who are looking up, and the blue prediction box represents the students who are currently looking down.

[0074] In one embodiment of the present application, it should be noted that the experimental hardware environment is a 12GB NXI GeForce RTX 3060 GPU, an AMD 5600X 6-Core CPU, and the parallel computing framework uses CUDA-11.4. The software operating environment uses the Window 10 operating system, and the programming uses Python-3.7.3 and Pytorch-1.9.1 deep learning frameworks, and PyCharm 2021 is used for compilation.

[0075] In the above-mentioned method for detecting and identifying student behavior at the back of the classroom based on deep learning, first, the student behavior image data at the back of the classroom is obtained; then, a behavior detection and recognition model is constructed based on the YOLOX-m target detection network, and the backbone feature extraction module, neck feature enhancement module and prediction head module of the YOLOX-m target detection network are optimized; then, the behavior detection and recognition model is trained based on the student behavior image data at the back of the classroom; finally, the actual classroom image to be identified is input into the trained behavior detection and recognition model, and the behavior detection and recognition result is output. In other words, by collecting images from the back of the classroom, optimizing the various modules of the existing target detection network, combining the attention mechanism and the deep separable convolution, it is possible to accurately detect and identify the two behaviors of students raising their heads and lowering their heads in class from the back. By detecting and identifying these two typical behaviors, it is possible to timely and effectively reflect the students' learning status, thereby helping teachers to accurately grasp the students' learning situation, better improve the classroom, thereby improving the efficiency of teaching and learning, and further facilitating intelligent classroom teaching.

[0076] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0077] Based on the same inventive concept, the embodiment of the present application also provides a device for detecting and identifying student behavior at the back of a classroom based on deep learning, which is used to implement the above-mentioned method for detecting and identifying student behavior at the back of a classroom based on deep learning. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more embodiments of the device for detecting and identifying student behavior at the back of a classroom based on deep learning provided below can be referred to the limitations of the method for detecting and identifying student behavior at the back of a classroom based on deep learning above, and will not be repeated here.

[0078] In one embodiment, Fig. 9 As shown, a device 900 for detecting and identifying student behavior at the back of a classroom based on deep learning is provided, comprising: a data acquisition module 901, a model building module 903, a model training module 905 and a behavior detection and identification module 907, wherein:

[0079] The data acquisition module 901 is used to acquire the behavior image data of students in the back of the classroom.

[0080] The model building module 903 is used to build a behavior detection and recognition model based on the YOLOX-m target detection network, and optimize the backbone feature extraction module, neck feature enhancement module and prediction head module of the YOLOX-m target detection network.

[0081] The model training module 905 is used to train the behavior detection and recognition model based on the student behavior image data at the back of the classroom.

[0082] The behavior detection and recognition module 907 is used to input the actual classroom image to be recognized into the trained behavior detection and recognition model and output the behavior detection and recognition result.

[0083] In one embodiment of the present application, the data acquisition module is further used to:

[0084] Mark students' behavior in class.

[0085] In one embodiment of the present application, the model building module is also used to:

[0086] A module based on the MSCA attention mechanism is added to each output layer of the backbone feature extraction module to improve the feature extraction capability of the behavior detection and recognition model.

[0087] In one embodiment of the present application, the model building module is also used to:

[0088] A depth-wise separable convolution minimum module DBS is constructed to reduce the number of parameters and the amount of calculation of the behavior detection and recognition model.

[0089] In one embodiment of the present application, the model building module is also used to:

[0090] The GIoU loss function is used instead of the IoU loss function to speed up the model fitting the position relationship between the predicted box and the real box.

[0091] In one embodiment of the present application, the method further includes:

[0092] The detection capability of the behavior detection and recognition model is evaluated using precision, recall, detection accuracy and average detection accuracy.

[0093] Each module in the above-mentioned device for detecting and identifying student behavior in the back of the classroom based on deep learning can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0094] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for detecting and identifying student behavior at the back of the classroom based on deep learning is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0095] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0096] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0097] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0098] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0099] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0100] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0101] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0102] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for detecting and identifying student behavior at the back of a classroom based on deep learning, characterized in that: The method comprises: Obtain image data of student behavior at the back of the classroom; Build a behavior detection and recognition model based on the YOLOX-m target detection network, and optimize the backbone feature extraction module, neck feature enhancement module and prediction head module of the YOLOX-m target detection network; Training the behavior detection and recognition model based on the behavior image data of students in the back of the classroom; The actual classroom image to be identified is input into the trained behavior detection and recognition model, and the behavior detection and recognition result is output.

2. According to the method for detecting and identifying student behavior at the back of the classroom based on deep learning in claim 1, it is characterized in that: The acquisition of student behavior image data at the back of the classroom includes: Mark students' behavior in class.

3. According to a method for detecting and identifying student behavior at the back of a classroom based on deep learning according to claim 1, it is characterized in that: The optimized backbone feature extraction module comprises: A module based on the MSCA attention mechanism is added to each output layer of the backbone feature extraction module to improve the feature extraction capability of the behavior detection and recognition model.

4. According to a method for detecting and identifying student behavior at the back of a classroom based on deep learning according to claim 1, it is characterized in that: The neck feature optimization and strengthening module comprises: A depth-wise separable convolution minimum module DBS is constructed to reduce the number of parameters and the amount of calculation of the behavior detection and recognition model.

5. According to a method for detecting and identifying student behavior at the back of a classroom based on deep learning according to claim 1, it is characterized in that: The optimization prediction head module includes: The GIoU loss function is used instead of the IoU loss function to speed up the model fitting the position relationship between the predicted box and the real box.

6. The method for detecting and identifying student behavior at the back of the classroom based on deep learning according to claim 1 is characterized in that: The method further comprises: The detection capability of the behavior detection and recognition model is evaluated using precision, recall, detection accuracy and average detection accuracy.

7. A device for detecting and identifying student behavior at the back of a classroom based on deep learning, characterized in that: The device comprises: A data acquisition module is used to acquire the image data of student behavior at the back of the classroom; A model building module is used to build a behavior detection and recognition model based on the YOLOX-m target detection network, and optimize the backbone feature extraction module, neck feature enhancement module and prediction head module of the YOLOX-m target detection network; A model training module, used for training the behavior detection and recognition model based on the behavior image data of students in the back of the classroom; The behavior detection and recognition module is used to input the actual classroom image to be recognized into the trained behavior detection and recognition model and output the behavior detection and recognition result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images