Intelligent recognition of students' classroom behavior and cognitive state association method in offline classrooms

Abstract

The present invention designs a method for intelligently identifying students' classroom behaviors and associating their cognitive states in offline classrooms. The method automatically analyzes students' classroom behaviors in classroom recordings through improved target extraction algorithms and target re-identification algorithms, accurately evaluates students' cognitive states by designing appropriate classroom pre-tests and post-tests, and finally uses time series modeling to personalize the association model between students' classroom behaviors and cognitive states. This method can significantly improve the processing efficiency of classroom video data, realize personalized association and analysis between cross-modal data, and has broad application prospects in the field of artificial intelligence education applications.

Description

Technical Field

[0001] The present invention designs a method for intelligently identifying students' classroom behaviors and associating their cognitive states in offline classrooms. The method automatically analyzes students' classroom behaviors in classroom recordings through improved target extraction algorithms and target re-identification algorithms, accurately evaluates students' cognitive states by designing appropriate classroom pre-tests and post-tests, and finally uses time series modeling to personalize the association model between students' classroom behaviors and cognitive states. This method can significantly improve the processing efficiency of classroom video data, realize personalized association and analysis between cross-modal data, and has broad application prospects in the field of artificial intelligence education applications. Background Art

[0002] With the development of artificial intelligence (AI) technology, applying it to education and teaching processes to promote the development of my country's education and improve teaching quality has become a hot topic in educational research. In the era of educational informatization 2.0, educational informatization has entered a new era of development. The large-scale application of emerging educational technologies such as mobile internet, big data, and artificial intelligence is actively promoting the establishment and improvement of smart campuses. Among the many applications of educational informatization, using AI technology to identify student classroom behavior and analyze its relationship with academic gains to improve personalized teaching is a worthy research topic.

[0003] Currently, most universities, primary and secondary schools in my country have installed classroom recording equipment, which can fully record the teaching process and can be used for subsequent research and analysis. However, in current teaching practices, due to the lack of professional data processing, mining, and analysis methods, students' classroom behaviors are simply detected manually through simple means such as attendance, raising hands in class, and sleeping, or students' classroom behavior is manually analyzed by reviewing recorded videos after class. This valuable resource of recorded videos is not fully utilized. On the other hand, although many scholars use offline classroom process evaluation to assess students' cognitive state, due to the high level of manual participation in cognitive state evaluation, only a small part of classroom teaching can be evaluated. Moreover, the lack of personalized association between cognitive state and students' classroom behavior makes it difficult to make a scientific evaluation of all classroom situations in colleges and universities.

[0004] Currently, research on student learning behavior is mostly empirically conducted through methods such as classroom observation, questionnaires, or scales. These methods suffer from poor real-time performance and require a large amount of manual work. With the development of deep learning technology, some scholars have begun using object detection algorithms to identify student behavior. However, due to factors such as the large number of student objects, severe mutual occlusion, imbalanced categories, and the difficulty in distinguishing between student behaviors in classrooms, identifying student behavior in complex classroom settings remains challenging. Furthermore, due to the lack of a direct correlation between student behavior and cognitive state, how to use student behavior to assess cognitive state remains a pressing challenge. Summary of the Invention

[0005] To address the above issues, the present invention uses a large number of available teaching videos to study a personalized correlation method for offline classroom students' classroom behavior and cognitive state. First, combined with video data from specific teaching scenarios, the paper explores the automatic identification of students' classroom behavior based on deep learning technology, focusing on solving the problem of identifying students in offline classrooms with complex and diverse scene contents. Secondly, a scientific and reasonable cognitive state assessment scheme is designed to achieve accurate assessment of students' cognitive state. Finally, relying on a time series model to correlate students' classroom behavior and cognitive state, the paper constructs personalized characteristics of students' classroom learning, helping teachers to grasp students' classroom status in a timely manner and effectively improve the quality of classroom teaching.

[0006] This paper addresses the challenges of insufficient intelligent recognition of student classroom behavior and a lack of research on the correlation between classroom behavior and cognitive status in the field of artificial intelligence education. This paper proposes a method for intelligently identifying student classroom behavior and correlating it with cognitive status in offline classrooms. The method is evaluated using real-world university classrooms. Experiments demonstrate that the proposed intelligent classroom behavior recognition algorithm effectively addresses the problem of identifying student classroom behavior in complex teaching scenarios. The proposed personalized correlation method for classroom behavior and cognitive status captures individual student characteristics, helping teachers to better understand student status and improve teaching quality.

[0007] The present invention includes four steps: intelligent identification of student classroom behavior, tracking and re-identification of student goals, assessment of student cognitive status, and personalized association of student behavior and cognitive status. The specific steps are as follows:

[0008] Step 1: Intelligent Recognition of Student Classroom Behavior. First, a student behavior classification dataset is constructed using smart classroom image data and object annotation tools. A neural network is then constructed based on a combination of the object recognition algorithm Faster R-CNN and the residual network ResNet50. The student classroom behavior dataset is used to enable the network to learn target features in classroom images. Finally, the trained model accurately predicts student target boxes and behavior categories in offline classroom images, which are characterized by imbalanced categories, numerous student targets, and significant mutual obscuration.

[0009] Step 2: Student target tracking and re-identification. The student target tracking and re-identification model combines target box similarity and histogram similarity methods to solve the problem of student target matching in recorded videos. This method uses the intelligent recognition results of student targets in classroom recorded video data as input and outputs the sequence characteristics of a single student's behavior throughout the class. This information includes "student name," "time," "location at the corresponding time," and "behavior at the corresponding time."

[0010] Step 3, student cognitive status assessment. The present invention evaluates students' learning cognitive status through their understanding and mastery of knowledge points. First, we design pre-tests and post-tests in advance based on the teaching content, and then obtain learners' pre-test and post-test data on knowledge points in a real classroom scene with reference to the set evaluation scheme. Finally, we complete the evaluation of learners' cognitive status with two evaluation dimensions, coarse granularity and fine granularity. Among them, the evaluation of the degree of mastery of each knowledge point is a fine-grained evaluation, and the answer questions are evaluated one by one through the knowledge points. Giving a simple numerical evaluation of the student status (such as the number of correct answers, etc.) is a coarse-grained evaluation. Through fine-grained and coarse-grained evaluations, the students' learning cognitive status is quantified, and the students' learning is evaluated from a macro-generalization and micro-specific level.

[0011] Step 4: Personalize the correlation between student behavior and cognitive status. Based on intelligent identification of student behavior and assessment of cognitive status, personalized correlation analysis is conducted from three perspectives: coarse-grained behavior-cognition correlation, fine-grained behavior-cognition correlation, and individual student behavior-cognition correlation. Visualizations such as pie charts, histograms, scatter plots, and polynomial regression plots are used to identify the correlation between behavior and cognition. The Pearson correlation coefficient is calculated to analyze the correlation between behavior and cognition. By studying abnormal behavior and cognition, the potential impact of behavior on cognition is analyzed. By tracking the behavior and cognition of the same student over a long period of time, the potential impact of behavioral changes on cognition is analyzed.

[0012] Furthermore, the specific implementation of step 1 is as follows:

[0013] Step 1.1: Dataset creation. We sorted out and determined the categories of student classroom behaviors. Using annotation tools, we labeled the extracted video frames with student target boxes and behavior categories. Each label recorded the coordinates of the four vertices on the target box and their category. We obtained a VOC standard annotation XML file corresponding to each classroom image, thereby constructing a classroom behavior classification dataset.

[0014] Step 1.2: Build a neural network for intelligent classroom behavior recognition based on a combination of a target recognition network and a behavior classification network. This task is divided into two steps: student target region detection and behavior category classification. This reduces the problem of missed targets due to the difficulty in distinguishing between student classroom behaviors. The target recognition network consists of four parts: a feature extraction layer, a region proposal layer, an interest pooling layer, and a bounding box regression and classification layer.

[0015] In step 1.3, the labeled classroom behavior classification dataset is used as the training set of the neural network. The prepared sample dataset is used to train the classroom behavior intelligent recognition neural network in step 1.2, and the data is iteratively input into the neural network training until convergence. For the classroom images of students to be tested, the trained neural network model parameters are loaded to obtain the test results. The student object is selected with the target box and the behavior category of the student is obtained.

[0016] Furthermore, in step 1.2, the feature extraction layer extracts features through the CNN network to extract the feature map of the original image; the region proposal layer generates a detection frame through the RPN network. The RPN network generates anchor points at the corresponding positions of the feature map according to a certain ratio. Each anchor point generates 9 candidate frames according to the aspect ratio of [1:2, 1:1, 2:1] and the scale of [128, 256, 512]. The interest pooling layer extracts feature maps of the same size from candidate frames of different sizes to obtain a feature vector of fixed length. Unlike the original object detection algorithm, the bounding box regression and classification layer only performs bounding box regression and does not classify student behavior.

[0017] The feature extraction layer uses the VGG16 network to extract features. The module contains 5 layers of convolutional pooling units. The first two layers include two continuously stacked convolutional units and one pooling unit, while the last three layers contain three continuously stacked convolutional units and one pooling unit.

[0018] The region proposal layer uses a 1×1 convolution kernel to perform a classification convolution operation on the feature map obtained by the feature extraction layer to predict the binary classification prediction value of the 9 candidate boxes generated on each feature point, which belongs to the foreground and background. At the same time, the regression convolution operation is performed in parallel through the 1×1 convolution kernel to output the four-dimensional coordinate information of the 9 candidate boxes on the current feature point. The interest pooling layer maps the candidate box to the position corresponding to the Roi on the feature map according to the input image, divides the mapped area into regions of the same size, and performs a maximum pooling operation on each small area. The region proposal layer uses the RPN network loss, which includes category loss and bounding box loss. The loss calculation process is as follows:

[0019]

[0020]

[0021]

[0022] where N reg is the size of the feature map, N cls is the number of Roi generated, t i ={t x ,t y ,t w,t h} is a set of vectors representing the predicted offset of the candidate box, It represents the actual offset relative to the true value, p i Indicates the probability that the predicted candidate box contains the target, It represents the true value label, the background box has a value of 0, and the target box has a value of 1. In this method, λ is taken as 10 to ensure that the classification and regression loss weights are as close as possible;

[0023] Furthermore, after pooling, the bounding box regression layer calculates more accurate bounding box coordinates through three fully connected layers.

[0024] Furthermore, in step 1.2, after the target detection network is trained, the behavior classification network is trained on this basis. The behavior classification network consists of 49 convolutional layers and 1 fully connected layer. The student behavior classification network based on Resnet50 is composed of convolution, regularization, activation function, and maximum pooling. After continuous residual convolution modules, the feature vector of size 1 is input into the fully connected layer FC through average pooling operation, and finally the corresponding category probability is predicted by softmax.

[0025] Furthermore, the specific implementation of step 2 is as follows:

[0026] Step 2.1: Extract the video frames into a sequence of images at a certain time interval. Use the trained student classroom behavior intelligent recognition model to perform intelligent student target recognition on the sequence images to obtain a list of student locations.

[0027] Step 2.2: Based on the first frame of the image, manually specify each student target information to form three dictionary lists with the student name as the key and the student location, image, and behavior as the value. This will serve as the basis for subsequent student target tracking and re-identification.

[0028] In step 2.3, a strategy combining target box similarity and histogram similarity is used to track and re-identify the student target, where the target box similarity is calculated using the IOU indicator.

[0029] Step 2.4, process the sequence frame images extracted from the video in step 2.3, update the position reference list, image reference list, and behavior reference list, and realize student target re-identification;

[0030] In step 2.5, after the image passes through step 2.4, a position benchmark list and a behavior benchmark list are obtained. Using the obtained benchmarks, an XML file for each student is obtained. The file includes "student name," "time," "position at the corresponding time," and "behavior at the corresponding time." Images with different frame numbers will have information written into the original student XML file after passing step 2.4, and ultimately each student's XML file will be output.

[0031] Furthermore, first obtain the student position list, and calculate the iou index of the student position and the position benchmark list in sequence. The iou index calculation formula is as follows:

[0032]

[0033] Where A1 is the area of ​​the intersection box and AU is the area of ​​the union box.

[0034] The calculation formula for A1 is as follows:

[0035] A1=W1×H1

[0036] Where W1 is the width of the intersection box, and H1 is the height of the intersection box. If the two boxes do not have a union, W1 = H1 = 0. The calculation formula is as follows:

[0037] W1=max[min(x2,x2′)-max(x1,x1′),0]

[0038] H1=max[min(y2,y2′)-max(y1,y1′),0]

[0039] The coordinates of the upper left corner of rectangle 1 are (x1, y1), and the coordinates of the lower right corner are (x2, y2). The coordinates of the upper left corner of rectangle 2 are (x1′, y1′), and the coordinates of the lower right corner are (x2′, y2′). The calculation formula for AU is as follows:

[0040] AU=w×h+w ′ ×h′

[0041] The width of rectangular frame 1 is w=x2-x1, and the height is h=y2-y1; the width of rectangular frame 2 is w′=x2′-x1′, and the height is h′=y2′-y1′.

[0042] When calculating the histogram similarity, first cut the student image through the student position list to obtain the image list, and then compare the histogram similarity with the image benchmark list in turn. The specific operation method is as follows:

[0043] Normalize the image to (256, 256) and split it into a list of 16 images. Use the histogram function to get the histogram. Use the histogram correlation comparison function compareHist to calculate the correlation value of the cut image, and get the average value to get the similarity. The histogram function divides the histogram by pixel points. If the input image is an RGB image, the histograms of different channels will be concatenated. The compareHist correlation comparison formula is as follows:

[0044]

[0045] Where H1 and H2 are the one-dimensional histogram vectors of the image, i is the position coordinate of the vector, and its value range is [0, N-1], where N is the length of the vector. Its average value is calculated as follows:

[0046]

[0047]

[0048] When calculating the fusion of target box similarity and histogram similarity, it is necessary to set the weight ratio of the IOU indicator, and fuse the IOU indicator calculated between the student position and the position benchmark with the histogram similarity calculated between the student image and the image benchmark. The fusion calculation method is as follows:

[0049] P=P1×q+P2(1-q)

[0050] Among them, P1 is the iou indicator, q is the weight ratio of the iou indicator, and P2 is the histogram similarity.

[0051] The student position is compared with the position benchmark in turn, and the student image is compared with the image benchmark in turn to obtain a fusion value list and set a threshold. If the maximum value of the fusion value list is greater than the threshold, the student position, image, and behavior are updated to the position benchmark, image benchmark, and behavior benchmark respectively through the maximum value position until the student position list and image list are compared in turn.

[0052] Furthermore, the specific implementation of step 3 is as follows:

[0053] Step 3.1: Divide the knowledge points according to the teaching content and design pre-tests and post-tests for the knowledge points. The knowledge points are evenly distributed throughout the entire class teaching period to analyze students' behavior and cognitive status in different time periods;

[0054] Step 3.2, conduct classroom evaluation based on pre-test and post-test;

[0055] Step 3.3: Calculate the students' cognitive status based on the assessment results. The students' mastery of each knowledge point is divided into four categories based on their pre-test and post-test answers: correct pre-test and incorrect post-test, correct pre-test and correct post-test, incorrect pre-test and incorrect post-test, and incorrect pre-test and correct post-test. Correct post-test answers indicate mastery.

[0056] Step 3.4 outputs fine-grained evaluation and coarse-grained evaluation. The fine-grained evaluation counts student behavior according to the specific time period when the assessment knowledge point is taught, and is used to construct the degree of correlation between behavior and cognition from a detailed level; the coarse-grained evaluation counts student behavior throughout the entire class process and comprehensively evaluates students together with the pre- and post-test results data.

[0057] Furthermore, the specific implementation of step 4 is as follows:

[0058] Step 4.1: Coarse-grained behavioral-cognitive correlation: Visualize the behaviors of abnormal and normal students using pie charts and histograms, compare their behaviors and cognitions, and analyze the differences between them. Abnormal students are divided into those with behavioral abnormalities and those with cognitive abnormalities. By analyzing the characteristics of abnormal students, we can identify the possible impact of behavior on cognition. Specifically, we analyze whether students' cognitive abnormalities are caused by abnormal classroom behavior and whether their abnormal behavior has an impact on their cognition.

[0059] Step 4.2: Fine-grained behavioral-cognitive correlation. Analyze the correlation between student behavior and cognition during the teacher's teaching of knowledge points. Divide the video clips recorded in the smart classroom into different segments according to the start and end times of the teacher's teaching of the knowledge points. Identify the student behavior in these segments and obtain the student behavior data corresponding to the knowledge points. The fine-grained cognitive assessment results of the students are the number of errors in the pre-test and correct answers in the post-test for a certain knowledge point. Use the student behavior data corresponding to the knowledge points and the student fine-grained cognitive assessment results as two variables for correlation analysis, and obtain scatter plots, polynomial regression plots, and Pearson correlation coefficient results. If the scatter plot shows that the scatter points approach a straight line and the absolute value of the Pearson correlation coefficient approaches 1, then the student's behavior and cognition have a linear correlation. If the polynomial regression plot shows that the scatter points approach a curve, then the student's behavior and cognition have a nonlinear correlation.

[0060] Step 4.3: Analyze students' personalized characteristics. Collect statistics on a student's behavior in multiple different classes, construct behavioral histograms for multiple different classes, record their cognition of each knowledge point, and analyze the personalized characteristics of the relationship between a single student's classroom behavior and cognition. On this basis, it helps to timely detect abnormal cognitive status of individual students through visual information in the video and take appropriate intervention measures.

[0061] Furthermore, the convolution kernels in the convolution units in the feature extraction layer are all 3×3, the step size is all 1, and the pooling units all use the 2×2 maximum pooling method.

[0062] Furthermore, 3×3 maximum pooling and 7×7 average pooling operations are used in the behavior classification network.

[0063] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows:

[0064] 1. Object recognition is divided into two steps: box optimization and object classification. This effectively solves the problem of unsatisfactory object recognition results caused by imbalanced student object categories and small differences between students' objects in real classroom scenarios, significantly improving object recognition performance.

[0065] 2. Comprehensively utilize features such as box overlap and histogram similarity to solve the problem of student target tracking and re-identification in the classroom;

[0066] 3. A personalized association model between student behavior and cognitive state based on time series was proposed, and a new scheme for evaluating cognitive state based on visual features was constructed. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is the overall flow chart of the present invention.

[0068] Figure 2 It is an invented flowchart for intelligent identification of students' classroom behavior.

[0069] Figure 3 It is a flow chart of student target tracking and re-identification of the present invention.

[0070] Figure 4 It is a flow chart of the student cognitive status assessment method of the present invention.

[0071] Figure 5 It is a pie chart of the correlation between student behavior and cognition. DETAILED DESCRIPTION

[0072] The following uses a real classroom scene in a smart classroom as an example and combines it with the accompanying drawings to further illustrate the technical solution of the present invention.

[0073] The technical solution adopted by the present invention is as follows: Step 1, intelligent recognition of student classroom behavior. Based on the improved FasterR-CNN algorithm, the student target and the corresponding behavior type of a single-frame image are accurately identified. Step 2, student target tracking and re-identification. On the basis of the intelligent recognition of student classroom behavior in a single-frame image, the tracking and re-identification of student targets between multiple frames of images are realized, and the classroom behavior sequence characteristics of each student are output in time series, among which student re-identification is mainly aimed at classroom scenes with rapid changes in camera angles. Step 3, assessment of student cognitive status. Design scientific and reasonable pre-tests and post-tests to evaluate students' mastery of classroom knowledge points and accurately assess students' cognitive status in various time periods in class. Step 4, personalized association between student behavior and cognitive status. Use a time series model to model the personalized association method between student behavior and cognitive status, focus on finding personalized behavioral characteristics when students are in poor cognitive status, and construct a new solution for assessing cognitive status based on visual features.

[0074] In the above steps, steps 1 and 2 realize the recognition of students' classroom behavior and the tracking and re-identification of students' goals between different sequence data, which can fully reflect the behavioral status of each student in each time period of the class. Step 3 evaluates the students' cognitive status in each time period of the class from the perspective of teaching evaluation. Step 4 is the focus of the present invention, which mainly realizes the personalized association between the student behavior obtained in steps 1 and 2 and the cognitive status obtained in step 3.

[0075] The specific implementation of step 1 includes the following sub-steps:

[0076] Step 1.1: Dataset Creation. Classify and identify student classroom behavior categories. Using annotation tools, label the extracted video frames with student target boxes and behavior categories. Each label records the coordinates of the four vertices on the target box and their category. This generates a VOC (PASCAL Visual Object Classes) standard annotation XML file corresponding to each classroom image. This is used to construct a classroom behavior classification dataset.

[0077] Step 1.2: Neural Network Construction. This involves building a neural network for intelligent classroom behavior recognition based on a combination of a target recognition network and a behavior classification network. This task is divided into two steps: detecting student target areas and classifying behavior categories. This reduces the risk of missed detections due to the difficulty in distinguishing between student behaviors.

[0078] The target recognition network includes four parts: Feature Extraction Layer, Region Proposal Layer, Interest Pooling Layer, and Bounding Box Regression Classification Layer. The feature extraction part extracts features through the CNN network and extracts the feature map of the original image. The region proposal layer generates a detection frame through the RPN network. The RPN network generates anchor points at the corresponding positions of the feature map according to a certain ratio. Each anchor point generates nine candidate frames according to the aspect ratio of [1:2, 1:1, 2:1] and the scale of [128, 256, 512]. The interest pooling layer extracts feature maps of the same size from candidate frames of different sizes to obtain a feature vector of fixed length. Unlike the original target detection algorithm, the bounding box regression classification layer in the present invention only performs bounding box regression and does not perform student behavior classification.

[0079] Furthermore, the feature extraction layer uses the VGG16 network to extract features. This module contains five layers of convolutional pooling units. The first two layers consist of two consecutively stacked convolutional units and one pooling unit, while the last three layers contain three consecutively stacked convolutional units and one pooling unit. The convolution kernels in these layers are all 3×3 with a stride of 1, and the pooling method used is a 2×2 maximum pooling method.

[0080] Furthermore, the region proposal layer uses a 1×1 convolution kernel to perform a classification convolution operation on the feature map obtained by the feature extraction layer to predict the binary classification prediction value of the foreground and background of each feature point to generate 9 candidate boxes; at the same time, the regression convolution operation is performed in parallel through the 1×1 convolution kernel to output the four-dimensional coordinate information of the 9 candidate boxes on the current feature point. The interest pooling layer maps the candidate box to the position corresponding to the Roi on the feature map based on the input image, divides the mapped area into areas of the same size, and performs a maximum pooling operation (max pooling) on ​​each small area. The region proposal layer uses the RPN network loss, which includes category loss and bounding box loss. The loss calculation process is as follows:

[0081]

[0082]

[0083]

[0084] where N reg is the size of the feature map, N cls is the number of Roi generated, t i ={t x ,t y ,t w,t h} is a set of vectors representing the predicted offset of the candidate box, It represents the actual offset relative to the true value, p i Indicates the probability that the predicted candidate box contains the target, It represents the true value label, the background box has a value of 0, and the target box has a value of 1. In this method, λ is taken as 10 to ensure that the loss weights of classification and regression are as close as possible.

[0085] Furthermore, after pooling, the bounding box regression layer calculates more accurate bounding box coordinates through three fully connected layers.

[0086] After the object detection network is trained, the behavior classification network is trained on this basis. The behavior classification network is composed of 49 convolutional layers and one fully connected layer. Based on the ResNet50 student classification network, this network performs convolution, regularization, activation functions, and max pooling. After a series of residual convolution modules, the feature vector of size 1 is input to the fully connected layer (FC) through average pooling. Finally, a softmax operation is performed to predict the corresponding category probability. This part uses 3×3 max pooling and 7×7 average pooling operations.

[0087] Step 1.3: Predict student classroom behavior. Use the labeled classroom behavior classification dataset as the training set for the network model. Use the prepared sample dataset to train the classroom behavior intelligent recognition model from Step 1.2. Repeatedly input this data into the network training until the model converges. For the student classroom image to be tested, load the trained model parameters, obtain the test results, use the target box to select the student object, and obtain the student's behavior category.

[0088] Furthermore, step 2 further includes the following sub-steps:

[0089] Step 2.1: Extract the video frames into a sequence of images at a certain time interval (e.g., 1 second). Use the trained student classroom behavior intelligent recognition model to perform intelligent student target recognition on the sequence of images to obtain a list of student locations.

[0090] Step 2.2: Based on the first frame of the image, manually specify each student target information to form three dictionary lists with the student name as the key and the student location, image, and behavior as the value. This will serve as the basis for subsequent student target tracking and re-identification.

[0091] In step 2.3, a strategy combining target box similarity and histogram similarity is used to track and re-identify students. Target box similarity is calculated using the IOU indicator. First, obtain the student position list. Then, calculate the IOU indicator for the student position and the position reference list in sequence. The IOU indicator calculation formula is as follows:

[0092]

[0093] Where A1 is the area of ​​the intersection box and AU is the area of ​​the union box.

[0094] The calculation formula for A1 is as follows:

[0095] A1=W1×H1

[0096] Where W1 is the width of the intersection box, and H1 is the height of the intersection box. If the two boxes do not have a union, W1 = H1 = 0. The calculation formula is as follows:

[0097] W1=max[min(x2,x2′)-max(x1,x1′),0]

[0098] H1=max[min(y2,y2′)-max(y1,y1′),0]

[0099] The calculation formula for achieving AU is as follows:

[0100] AU=w×h+w ′ ×h′

[0101] The meaning of the specific data is shown in Table 1.

[0102] Table 1iou calculation data

[0103]

[0104] When calculating the histogram similarity, first cut the student image through the student position list to obtain the image list, and then compare the histogram similarity with the image benchmark list in turn. The specific operation method is as follows:

[0105] Normalize the image to (256, 256) and split it into a list of 16 images. Use the histogram function to get the histogram. Use the histogram correlation comparison function compareHist to calculate the correlation value of the cut images, and get the average value to get the similarity. The histogram function divides the histogram by pixel points. If the input image is an RGB image, the histograms of different channels will be concatenated. The compareHist correlation comparison formula is as follows:

[0106]

[0107] Where H1 and H2 are the one-dimensional histogram vectors of the image, i is the position coordinate of the vector, and its value range is [0, N-1], where N is the length of the vector. Its average value is calculated as follows:

[0108]

[0109]

[0110] When calculating the fusion of target box similarity and histogram similarity, it is necessary to set the weight ratio of the IOU indicator, and fuse the IOU indicator calculated between the student position and the position benchmark with the histogram similarity calculated between the student image and the image benchmark. The fusion calculation method is as follows:

[0111] P=P1×q+P2(1-q)

[0112] Among them, P1 is the iou indicator, q is the weight ratio of the iou indicator, and P2 is the histogram similarity.

[0113] The student position is compared with the position benchmark in turn, and the student image is compared with the image benchmark in turn to obtain a fusion value list. The threshold is set to 0.8. If the maximum value of the fusion value list is greater than the threshold, the student position, image, and behavior are updated to the position benchmark, image benchmark, and behavior benchmark respectively through the maximum value position until the student position list and image list are compared in turn.

[0114] Step 2.4: Perform step 2.3 on the sequence frame images extracted from the video, update the position reference list, image reference list, and behavior reference list, and realize student target re-identification;

[0115] In step 2.5, after the image passes through step 2.4, a position benchmark list and a behavior benchmark list are obtained. Using the obtained benchmarks, an XML file for each student is obtained. The file includes "student name," "time," "position at the corresponding time," and "behavior at the corresponding time." Images with different frame numbers will have information written into the original student XML file after passing step 2.4, and ultimately each student's XML file will be output.

[0116] Furthermore, step 3 further includes the following sub-steps:

[0117] Step 3.1: Divide the knowledge points according to the teaching content and design pre-tests and post-tests for each knowledge point. The knowledge points should be evenly distributed throughout the entire class teaching period to analyze students' behavior and cognitive status at different time periods.

[0118] Step 3.2, conduct classroom evaluation based on pre-test and post-test;

[0119] Step 3.3: Calculate the students' cognitive status based on the assessment results. The students' mastery of each knowledge point is divided into four categories based on their pre-test and post-test answers: correct pre-test and incorrect post-test, correct pre-test and correct post-test, incorrect pre-test and incorrect post-test, and incorrect pre-test and correct post-test. Correct post-test answers indicate mastery.

[0120] Step 3.4 outputs fine-grained and coarse-grained evaluations. Fine-grained evaluations measure student behavior characteristics based on the specific time period during which the assessed knowledge point is taught, used to establish the correlation between behavior and cognition at a detailed level. Coarse-grained evaluations measure student behavior throughout the entire class and, combined with pre- and post-test data, provide a comprehensive evaluation of the students.

[0121] Furthermore, step 4 further includes the following sub-steps:

[0122] Step 4.1, coarse-grained behavior-cognition correlation. Visualize the behavior of abnormal students and normal students through pie charts and histograms, compare the behavior and cognition between the two, and analyze the differences between the two. Abnormal students are divided into students with abnormal behavior and students with abnormal cognition. Students with abnormal behavior are students whose behavior or the total number of behaviors in a class is particularly prominent, or in a certain period of time, most students maintain a certain behavior (such as lowering their head), while only this student maintains another or several behaviors (such as raising their head or tilting their head). Students with abnormal cognition are students with outstanding post-class test scores. Whether the students' cognitive abnormality is caused by abnormal classroom behavior, and whether the students' abnormal behavior has a greater impact on the students' cognition. By analyzing the characteristics of abnormal students, find the possible impact of behavior on cognition.

[0123] Step 4.2: Fine-grained behavioral-cognitive correlation. Analyze the correlation between student behavior and cognition during the teacher's presentation of knowledge points. Divide the video clips recorded in the smart classroom into different segments based on the start and end times of the teacher's presentation of the knowledge points. Identify the student behavior in these segments and obtain the student behavior data corresponding to the knowledge points. The student's fine-grained cognitive assessment results are the number of errors in the pre-test and correct answers in the post-test for a particular knowledge point. Correlation analysis is performed using the student behavior data corresponding to the knowledge point and the student's fine-grained cognitive assessment results as two variables. Results such as scatter plots, polynomial regression plots, and Pearson correlation coefficients are obtained. If the scatter plot shows that the scatter points approach a straight line and the absolute value of the Pearson correlation coefficient approaches 1, then there is a linear correlation between the student's behavior and cognition. If the polynomial regression plot shows that the scatter points approach a curve, then there is a nonlinear correlation between the student's behavior and cognition.

[0124] Step 4.3: Analyze individual student characteristics. By analyzing a student's behavior across multiple different classes, we construct a behavioral histogram, recording their learning progress for each knowledge point. This allows us to analyze the personalized characteristics of each student's classroom behavior and cognitive relationships. This helps us identify cognitive abnormalities in individual students through visual information in the video and implement appropriate interventions.

[0125] The specific embodiments described herein are merely illustrative of the spirit of the present invention. Persons skilled in the art may make various modifications, additions, or substitutions to the described specific embodiments without departing from the spirit of the present invention or exceeding the scope of the appended claims.

Claims

1. A method for intelligently identifying students' classroom behaviors and associating their cognitive states in offline classrooms, characterized by: The steps include: Step 1: Intelligent recognition of student classroom behavior. First, we use smart classroom image data and object annotation tools to construct a student behavior classification dataset. We then build a neural network based on the Faster R-CNN object recognition algorithm and the ResNet50 residual network. We use the neural network to analyze the object features in the classroom images in the student classroom behavior dataset. Finally, we use the trained model to accurately predict student target boxes and behavior categories in offline classroom images. Step 2: Student target tracking and re-identification: Based on the intelligent recognition of student classroom behavior in a single-frame image, the system tracks and re-identifies student targets across multiple frames, outputting the sequence features of each student's classroom behavior in a time series. The output is the sequence features of a single student's behavior throughout the entire class. The output information includes the student's name, time, location at the corresponding time, and behavior at the corresponding time. Step 3: Assessing students' cognitive status. First, pre-tests and post-tests are designed based on the course content. Then, pre-test and post-test data on the learners' knowledge points are obtained in real classroom scenarios. Finally, the learners' cognitive status is assessed using two evaluation dimensions: coarse-grained and fine-grained. The assessment of the mastery of each knowledge point is a fine-grained evaluation, while the evaluation of the answer questions is performed one-to-one based on the knowledge points. The overall numerical evaluation of the student's status is a coarse-grained evaluation. Through both fine-grained and coarse-grained evaluations, the students' cognitive status is quantified, and their learning is evaluated from both a macro-generalized and micro-specific perspective. Step 4: Personalized association between student behavior and cognitive status. Based on the intelligent identification of student behavior and the assessment of cognitive status, personalized association analysis is conducted from three perspectives: coarse-grained behavior and cognition association, fine-grained behavior and cognition association, and individual student behavior and cognition association. Through visualization processing, the association between behavior and cognition is obtained, and the degree of correlation between behavior and cognition is analyzed by calculating the Pearson correlation coefficient.

2. The method for intelligently identifying students' classroom behaviors and associating their cognitive states in an offline classroom as claimed in claim 1, characterized in that: The specific implementation of step 1 is as follows: Step 1.1: Dataset creation. We sorted out and determined the categories of student classroom behaviors. Using annotation tools, we labeled the extracted video frames with student target boxes and behavior categories. Each label recorded the coordinates of the four vertices on the target box and their category. We obtained a VOC standard annotation XML file corresponding to each classroom image, thereby constructing a classroom behavior classification dataset. Step 1.2: Build a neural network for intelligent classroom behavior recognition based on a combination of a target recognition network and a behavior classification network. This task is divided into two steps: detecting student target areas and classifying behavior categories. This reduces the problem of missed targets due to the difficulty in distinguishing between student behaviors. The target recognition network consists of four parts: feature extraction layer, region proposal layer, interest pooling layer, and bounding box regression and classification layer. In step 1.3, the labeled classroom behavior classification dataset is used as the training set of the neural network. The prepared sample dataset is used to train the classroom behavior intelligent recognition neural network in step 1.2, and the data is iteratively input into the neural network training until convergence. For the classroom images of students to be tested, the trained neural network model parameters are loaded to obtain the test results. The student object is selected with the target box and the behavior category of the student is obtained.

3. The method for intelligently identifying students' classroom behaviors and associating their cognitive states in an offline classroom as claimed in claim 2, characterized in that: In step 1.2, the feature extraction layer extracts features through the CNN network and extracts the feature map of the original image; the region proposal layer generates a detection frame through the RPN network. The RPN network generates anchor points at the corresponding positions of the feature map according to a certain ratio. Each anchor point generates 9 candidate frames according to the aspect ratio of [1:2, 1:1, 2:1] and the scale of [128, 256, 512]. The interest pooling layer extracts feature maps of the same size from candidate frames of different sizes and obtains a feature vector of fixed length. Unlike the original object detection algorithm, the bounding box regression and classification layer only performs bounding box regression and does not classify student behavior. The feature extraction layer uses the VGG16 network to extract features. The module contains 5 layers of convolutional pooling units. The first two layers include two continuously stacked convolutional units and one pooling unit, while the last three layers include three continuously stacked convolutional units and one pooling unit. The region proposal layer uses a 1×1 convolution kernel to perform a classification convolution operation on the feature map obtained by the feature extraction layer to predict the binary classification prediction value of the foreground and background for each feature point. At the same time, the regression convolution operation is performed in parallel through the 1×1 convolution kernel to output the four-dimensional coordinate information of the 9 candidate boxes on the current feature point. The interest pooling layer maps the candidate box to the position corresponding to the Roi on the feature map according to the input image, divides the mapped area into regions of the same size, and performs a maximum pooling operation on each small area. The region proposal layer uses the RPN network loss, including category loss and bounding box loss. The loss calculation process is as follows: where N reg is the size of the feature map, N cls is the number of Roi generated, t i ={t x ,t y ,t w ,t h } is a set of vectors representing the predicted offset of the candidate box, It represents the actual offset relative to the true value, p i Indicates the probability that the predicted candidate box contains the target, It represents the true value label, the background box has a value of 0, the target box has a value of 1, and λ is the weight coefficient; further, after pooling, the bounding box regression layer calculates more accurate bounding box coordinates through three fully connected layers.

4. The method for intelligently identifying students' classroom behaviors and associating their cognitive states in an offline classroom as claimed in claim 2, characterized in that: In step 1.2, after the target detection network is trained, the behavior classification network is trained on this basis. The behavior classification network consists of 49 convolutional layers and 1 fully connected layer. The student behavior classification network based on Resnet50 is composed of convolution, regularization, activation function, and maximum pooling. After continuous residual convolution modules, the feature vector of size 1 is input into the fully connected layer FC through average pooling operation, and finally the corresponding category probability is predicted by softmax.

5. The method for intelligently identifying students' classroom behaviors and associating their cognitive states in an offline classroom as claimed in claim 2, characterized in that: The specific implementation of step 2 is as follows: Step 2.1: Extract the video frames into a sequence of images at a certain time interval. Use the trained student classroom behavior intelligent recognition model to perform intelligent student target recognition on the sequence images to obtain a list of student locations. Step 2.2: Based on the first frame of the image, manually specify each student target information to form three dictionary lists with the student name as the key and the student location, image, and behavior as the value. This will serve as the basis for subsequent student target tracking and re-identification. In step 2.3, a strategy combining target box similarity and histogram similarity is used to track and re-identify the student target, where the target box similarity is calculated using the IOU indicator. Step 2.4, process the sequence frame images extracted from the video in step 2.3, update the position reference list, image reference list, and behavior reference list, and realize student target re-identification; In step 2.5, after the image passes through step 2.4, a position benchmark list and a behavior benchmark list are generated. Using these benchmarks, an XML file is generated for each student. The file includes "student name," "time," "position at the corresponding time," and "behavior at the corresponding time." After passing step 2.4, images with different frame rates will have their information written into the original student XML file, ultimately resulting in each student's XML file as output.

6. The method for intelligently identifying students' classroom behaviors and associating their cognitive states in an offline classroom as claimed in claim 5, characterized in that: First, obtain the student position list, and calculate the iou indicator for the student position and the position benchmark list in turn. The iou indicator calculation formula is as follows: Where A1 is the area of ​​the intersection box, and AU is the area of ​​the union box; The calculation formula for A1 is as follows: A1=W1×H1 Where W1 is the width of the intersection box, and H1 is the height of the intersection box. If the two boxes do not have a union, W1 = H1 = 0. The calculation formula is as follows: W1=max[min(x2,x2')max(x1,x1'),0] H1=max[min(y2,y2')-max(y1,y1'),0] The coordinates of the upper left corner of rectangle 1 are (x1, y1), and the coordinates of the lower right corner are (x2, y2). The coordinates of the upper left corner of rectangle 2 are (x1′, y1′), and the coordinates of the lower right corner are (x2′, y2′). The calculation formula for AU is as follows: AU=w×h+w'×h' Among them, the width of rectangular frame 1 is w=x2-x1, and the height is h=y2-y1; the width of rectangular frame 2 is w′=x2′-x1′, and the height is h′=y2′-y1′; When calculating the histogram similarity, first cut the student image through the student position list to obtain the image list, and then compare the histogram similarity with the image benchmark list in turn. The specific operation method is as follows: Normalize the image to (256, 256) and split it into a list of 16 images. Use the histogram function to get the histogram. Use the histogram correlation comparison function compareHist to calculate the correlation value of the cut image, and get the average value to get the similarity. The histogram function divides the histogram by pixel points. If the input image is an RGB image, the histograms of different channels will be concatenated. The compareHist correlation comparison formula is as follows: Where H1 and H2 are the one-dimensional histogram vectors of the image, i is the position coordinate of the vector, and its value range is [0, N-1], where N is the length of the vector. The average value is calculated as follows: When calculating the fusion of target box similarity and histogram similarity, it is necessary to set the weight ratio of the IOU indicator, and fuse the IOU indicator calculated between the student position and the position benchmark with the histogram similarity calculated between the student image and the image benchmark. The fusion calculation method is as follows: P=P1×q+P2(1-q) Among them, P1 is the iou indicator, q is the weight ratio of the iou indicator, and P2 is the histogram similarity; The student position is compared with the position benchmark in turn, and the student image is compared with the image benchmark in turn to obtain a fusion value list and set a threshold. If the maximum value of the fusion value list is greater than the threshold, the student position, image, and behavior are updated to the position benchmark, image benchmark, and behavior benchmark respectively through the maximum value position until the student position list and image list are compared in turn.

7. The method for intelligently identifying students' classroom behaviors and associating their cognitive states in an offline classroom as claimed in claim 1, characterized in that: The specific implementation of step 3 is as follows: Step 3.1: Divide the knowledge points according to the teaching content and design pre-tests and post-tests for the knowledge points. The knowledge points are evenly distributed throughout the entire class teaching period to analyze students' behavior and cognitive status in different time periods; Step 3.2, conduct classroom evaluation based on pre-test and post-test; Step 3.3: Calculate the students' cognitive status based on the assessment results. Divide the students' mastery of each knowledge point into four categories based on their pre-test and post-test answers: correct pre-test and incorrect post-test, correct pre-test and correct post-test, incorrect pre-test and incorrect post-test, and incorrect pre-test and correct post-test. Correct post-test answers indicate mastery. Step 3.4 outputs fine-grained evaluation and coarse-grained evaluation. The fine-grained evaluation counts student behavior according to the specific time period when the assessment knowledge point is taught, and is used to construct the degree of correlation between behavior and cognition from a detailed level; the coarse-grained evaluation counts student behavior throughout the entire class process and uses the pre- and post-test results data to conduct a comprehensive evaluation of the students.

8. The method for intelligently identifying students' classroom behaviors and associating their cognitive states in an offline classroom as claimed in claim 7, characterized in that: The specific implementation of step 4 is as follows: Step 4.1: Coarse-grained behavioral-cognitive correlation: Visualize the behaviors of abnormal and normal students using pie charts and histograms, compare their behaviors and cognitions, and analyze the differences between them. Abnormal students are divided into those with behavioral abnormalities and those with cognitive abnormalities. By analyzing the characteristics of abnormal students, we can identify the possible impact of behavior on cognition. Specifically, we analyze whether students' cognitive abnormalities are caused by abnormal classroom behavior and whether their abnormal behavior has an impact on their cognition. Step 4.2: Fine-grained behavioral-cognitive correlation. Analyze the correlation between student behavior and cognition during the teacher's teaching of knowledge points. Divide the video clips recorded in the smart classroom into different segments according to the start and end times of the teacher's teaching of the knowledge points. Identify the student behavior in these segments and obtain the student behavior data corresponding to the knowledge points. The fine-grained cognitive assessment results of the students are the number of errors in the pre-test and correct answers in the post-test for a certain knowledge point. Use the student behavior data corresponding to the knowledge points and the student fine-grained cognitive assessment results as two variables for correlation analysis, and obtain scatter plots, polynomial regression plots, and Pearson correlation coefficient results. If the scatter plot shows that the scatter points approach a straight line and the absolute value of the Pearson correlation coefficient approaches 1, then the student's behavior and cognition have a linear correlation. If the polynomial regression plot shows that the scatter points approach a curve, then the student's behavior and cognition have a nonlinear correlation. Step 4.3, analysis of students’ personalized characteristics; Statistics are collected on a student's situations in multiple different classes, and behavioral histograms of these students in different classes are constructed to record their cognition of each knowledge point. The personalized characteristics of the relationship between a single student's classroom behavior and cognition are analyzed. On this basis, it helps to timely discover abnormal cognitive states of individual students through visual information in the video and take appropriate intervention measures.

9. The method for intelligently identifying students' classroom behaviors and associating their cognitive states in an offline classroom as claimed in claim 3, characterized in that: The convolution kernels in the convolution units in the feature extraction layer are all 3×3, the step size is all 1, and the pooling units all use the 2×2 maximum pooling method.

10. The method for intelligently identifying students' classroom behaviors and associating their cognitive states in an offline classroom as claimed in claim 4, characterized in that: The behavior classification network uses 3×3 maximum pooling and 7×7 average pooling operations.

Images