Classroom concentration assessment method and device integrating behavioral analysis and sentiment analysis

Abstract

The present invention relates to the field of computer vision technology, and in particular to a method and device for evaluating classroom concentration that integrates behavior analysis and sentiment analysis. The method for evaluating classroom concentration that integrates behavior analysis and sentiment analysis includes: collecting data on the student's classroom status through a camera to obtain a training data set; using the training data set to train a two-stage classroom behavior recognition model to be trained and a classroom emotion recognition model to be trained to obtain a two-stage classroom behavior recognition model and an emotion recognition model; obtaining the current state data set for recognition to obtain student behavior recognition results and student emotion recognition results; quantifying concentration through a fuzzy comprehensive evaluation method based on a hierarchical analysis method to obtain a concentration evaluation result. The present invention is an automatic and efficient classroom concentration evaluation method that takes into account both behavior analysis and sentiment analysis.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a classroom concentration assessment method and device integrating behavior analysis and sentiment analysis. Background Art

[0002] With the continuous development of informatization, the way of acquiring and imparting knowledge has undergone revolutionary changes, and the digital reform in the field of education has been gradually accelerated. In the context of my country's continuous promotion of educational informatization and intelligence, it has become increasingly important to establish a complete classroom monitoring and analysis system and conduct in-depth research on classroom teaching activities.

[0003] Classroom teaching, as a basic form of teaching, has always been the core of educational research. In the classroom learning environment, teachers get feedback by observing students' behavior and emotional expressions. This human-dominated detection method is difficult to avoid subjectivity, and has problems such as low efficiency and incomplete observation, making it difficult to expand to intelligent information-based classroom teaching.

[0004] Based on computer vision technology, analyzing classroom pictures or videos and returning student status scores can save a lot of energy, freeing teachers from judging student status based on images or videos, and directly adjusting teaching strategies based on the returned scores, thereby improving classroom learning effects. However, most of the current computer vision-based methods are behavioral analysis of students or emotional analysis of students, which cannot take into account both behavior and emotion, which has become a bottleneck for improving classroom learning effects. In addition, in classroom student behavior recognition, the same method is usually used to identify all student behaviors, which may lead to neglect of behavioral differences. Most existing behavior recognition models are based on RGB or skeleton images for behavior recognition. The RGB modality is rich in appearance information but is greatly affected by lighting and background changes, while the skeleton modality is robust to background changes and focuses on human skeleton movement but lacks appearance information of people and objects.

[0005] In the existing technology, there is a lack of an automatic and efficient classroom concentration assessment method that takes into account both behavioral analysis and sentiment analysis. Summary of the invention

[0006] In order to solve the technical problem that the existing technology cannot take into account both behavior analysis and sentiment analysis, the embodiment of the present invention provides a classroom concentration assessment method and device that integrates behavior analysis and sentiment analysis. The technical solution is as follows:

[0007] On the one hand, a method for evaluating classroom concentration by integrating behavior analysis and sentiment analysis is provided, the method being implemented by a classroom concentration evaluation device, and the method comprising:

[0008] Collect data on students' classroom status through cameras to obtain a training data set; the training data set includes a behavior recognition data set and an emotion recognition data set;

[0009] A two-stage classroom behavior recognition model to be trained is established based on the residual neural network structure and the Yo l ov8 model structure; a classroom emotion recognition model to be trained is established based on the residual neural network structure of the attention mechanism;

[0010] Using the behavior recognition data set, training the two-stage classroom behavior recognition model to be trained to obtain a two-stage classroom behavior recognition model;

[0011] Using the emotion recognition data set, training the classroom emotion recognition model to be trained to obtain an emotion recognition model;

[0012] Acquire a current student classroom status video; perform data processing on the current student classroom status video to obtain a current status data set; input the current status data set into the two-stage classroom behavior recognition model to obtain a student behavior recognition result; input the current status data into the emotion recognition model to obtain a student emotion recognition result;

[0013] According to the student behavior recognition results and the student emotion recognition results, concentration is quantified by a fuzzy comprehensive evaluation method based on the hierarchical analysis method to obtain a concentration evaluation result.

[0014] Wherein, the two-stage classroom behavior recognition model includes a large-amplitude behavior recognition model and a small-amplitude behavior recognition model;

[0015] The large-amplitude behavior recognition model includes a data fusion layer, a feature fusion layer and a linear output layer.

[0016] Optionally, collecting data on the student's classroom status through a camera to obtain a training data set includes:

[0017] Using cameras to shoot different angles of different classrooms to obtain monitoring videos; performing frame processing on the monitoring videos to obtain classroom learning pictures;

[0018] Based on the classroom learning pictures, the OpenPose algorithm is used to estimate the human body posture and obtain the coordinates of the key points of the students' bodies;

[0019] Generate a picture according to the coordinates of the key points of the student's body to obtain a student RGB picture and a student skeleton picture;

[0020] Based on the student's head-lowering, head-raising and head-sideways behavior movements, the student RGB image and the student skeleton image are screened to obtain a large-amplitude behavior recognition data set; the large-amplitude behavior recognition data set is used for training a large-amplitude behavior recognition model;

[0021] Based on the students' mobile phone playing, note-taking and empty-handed behavior movements, Label Img software is used to annotate the student RGB image and the student skeleton image to obtain a small amplitude behavior recognition data set; the small amplitude behavior recognition data set is used for training a small amplitude behavior recognition model;

[0022] According to the student RGB image, the student face area image is cropped by a preset face detection algorithm to obtain a student face picture; based on the students' facial expressions of joy, expressionlessness, fatigue, distraction and confusion, the student face picture is screened to obtain an emotion recognition data set; the emotion recognition data set is used for training the emotion recognition model.

[0023] Optionally, the two-stage classroom behavior recognition model includes a large-amplitude behavior recognition model, and the step of inputting the current state data set into the two-stage classroom behavior recognition model to obtain a student behavior recognition result includes:

[0024] Dividing the current state data set to obtain a current large-scale behavior recognition data set;

[0025] Inputting the current large-scale behavior recognition data set into the data fusion layer to obtain a fused RGB image and a fused skeleton image;

[0026] Inputting the fused RGB image and the fused skeleton image into the feature fusion layer to obtain fused features;

[0027] The fused features are input into the linear output layer for data processing to obtain a large-scale behavior recognition result.

[0028] Optionally, the step of inputting the current large-scale behavior recognition data set into the data fusion layer to obtain a fused RGB image and a fused skeleton image includes:

[0029] Preprocessing the current large-scale behavior recognition data set to obtain a first RGB image and a first skeleton image;

[0030] Resize the first RGB image and the first skeleton image to obtain a second RGB image and a second skeleton image;

[0031] Using a weighted fusion method, the second skeleton image is integrated into the second RGB image to obtain a fused RGB image;

[0032] The second RGB image is integrated into the second skeleton image by using a weighted fusion method to obtain a fused skeleton image.

[0033] Optionally, inputting the fused RGB image and the fused skeleton image into the feature fusion layer to obtain fused features includes:

[0034] Inputting the fused RGB image and the fused skeleton image into the feature fusion layer for feature extraction to obtain a first RGB feature image and a first skeleton feature image;

[0035] Resize the first RGB feature map and the first skeleton feature map to obtain a second RGB feature map and a second skeleton feature map;

[0036] Calculating according to the second RGB image and the second skeleton feature image to obtain a low-order similarity matrix;

[0037] Calculating according to the low-order similarity matrix to obtain a high-order similarity matrix;

[0038] Calculate according to the high-order similarity matrix to obtain a row vector and a column vector;

[0039] Weighted fusion is performed according to the row vector and the column vector to obtain a fusion feature.

[0040] Optionally, the concentration evaluation result is obtained by quantifying the concentration according to the student behavior recognition result and the student emotion recognition result through a fuzzy comprehensive evaluation method based on the hierarchical analysis method, including:

[0041] Determine evaluation factors according to evaluation criteria for student behavior; the evaluation criteria include positive behavior, negative behavior, positive emotion and negative emotion; the evaluation factors include first-level evaluation factors and second-level evaluation factors; the first-level evaluation factors are a combination of behavior and emotion in the evaluation criteria; the first-level evaluation factors represent the behavior-emotion state of the student; the second-level evaluation factors refer to the behavior actions and emotion representations corresponding to the behavior-emotion state;

[0042] Setting a concentration evaluation level; assigning a value to the concentration evaluation level to obtain a concentration level value; the concentration evaluation level includes low, lower, medium, higher and high;

[0043] The analytic hierarchy process is used to assign weights to the first-level evaluation factors and the second-level evaluation privacy to obtain the evaluation factor weights;

[0044] Establish a single factor evaluation matrix based on the preset questionnaire survey results;

[0045] Based on the first-level evaluation factors and the second-level evaluation factors, according to the single-factor evaluation matrix and the evaluation factor weights, a fuzzy comprehensive evaluation is performed on the student behavior recognition results and the student emotion recognition results to obtain a concentration evaluation result.

[0046] On the other hand, a classroom concentration device integrating behavior analysis and sentiment analysis is provided, and the device is applied to a classroom concentration evaluation method integrating behavior analysis and sentiment analysis, and the device includes:

[0047] A training data set acquisition module is used to collect data on the student's classroom status through a camera to obtain a training data set; the training data set includes a behavior recognition data set and an emotion recognition data set;

[0048] Model building module, used to establish a two-stage classroom behavior recognition model to be trained based on the residual neural network structure and the Yo l ov8 model structure; to establish a classroom emotion recognition model to be trained based on the residual neural network structure of the attention mechanism;

[0049] A behavior recognition model training module is used to use the behavior recognition data set to train the two-stage classroom behavior recognition model to be trained to obtain a two-stage classroom behavior recognition model;

[0050] An emotion recognition model training module is used to use the emotion recognition data set to train the classroom emotion recognition model to be trained to obtain an emotion recognition model;

[0051] The student status recognition module is used to obtain the current student classroom status video; perform data processing on the current student classroom status video to obtain a current status data set; input the current status data set into the two-stage classroom behavior recognition model to obtain a student behavior recognition result; input the current status data into the emotion recognition model to obtain a student emotion recognition result;

[0052] The concentration evaluation module is used to quantify the concentration according to the student behavior recognition result and the student emotion recognition result through a fuzzy comprehensive evaluation method based on the hierarchical analysis method to obtain the concentration evaluation result.

[0053] Wherein, the two-stage classroom behavior recognition model includes a large-amplitude behavior recognition model and a small-amplitude behavior recognition model;

[0054] The large-amplitude behavior recognition model includes a data fusion layer, a feature fusion layer and a linear output layer.

[0055] Optionally, the training data set acquisition module is further used to:

[0056] Using cameras to shoot different angles of different classrooms to obtain monitoring videos; performing frame processing on the monitoring videos to obtain classroom learning pictures;

[0057] Based on the classroom learning pictures, the OpenPose algorithm is used to estimate the human body posture and obtain the coordinates of the key points of the students' bodies;

[0058] Generate a picture according to the coordinates of the key points of the student's body to obtain a student RGB picture and a student skeleton picture;

[0059] Based on the student's head-lowering, head-raising and head-sideways behavior movements, the student RGB image and the student skeleton image are screened to obtain a large-amplitude behavior recognition data set; the large-amplitude behavior recognition data set is used for training a large-amplitude behavior recognition model;

[0060] Based on the students' mobile phone playing, note-taking and empty-handed behavior movements, Label Img software is used to annotate the student RGB image and the student skeleton image to obtain a small amplitude behavior recognition data set; the small amplitude behavior recognition data set is used for training a small amplitude behavior recognition model;

[0061] According to the student RGB image, the student face area image is cropped by a preset face detection algorithm to obtain a student face picture; based on the students' facial expressions of joy, expressionlessness, fatigue, distraction and confusion, the student face picture is screened to obtain an emotion recognition data set; the emotion recognition data set is used for training the emotion recognition model.

[0062] Optionally, the student status identification module is further used to:

[0063] Dividing the current state data set to obtain a current large-scale behavior recognition data set;

[0064] Inputting the current large-scale behavior recognition data set into the data fusion layer to obtain a fused RGB image and a fused skeleton image;

[0065] Inputting the fused RGB image and the fused skeleton image into the feature fusion layer to obtain fused features;

[0066] The fused features are input into the linear output layer for data processing to obtain a large-scale behavior recognition result.

[0067] Optionally, the student status identification module is further used to:

[0068] Preprocessing the current large-scale behavior recognition data set to obtain a first RGB image and a first skeleton image;

[0069] Resize the first RGB image and the first skeleton image to obtain a second RGB image and a second skeleton image;

[0070] Using a weighted fusion method, the second skeleton image is integrated into the second RGB image to obtain a fused RGB image;

[0071] The second RGB image is integrated into the second skeleton image by using a weighted fusion method to obtain a fused skeleton image.

[0072] Optionally, the student status identification module is further used to:

[0073] Inputting the fused RGB image and the fused skeleton image into the feature fusion layer for feature extraction to obtain a first RGB feature image and a first skeleton feature image;

[0074] Resize the first RGB feature map and the first skeleton feature map to obtain a second RGB feature map and a second skeleton feature map;

[0075] Calculating according to the second RGB image and the second skeleton feature image to obtain a low-order similarity matrix;

[0076] Calculating according to the low-order similarity matrix to obtain a high-order similarity matrix;

[0077] Calculate according to the high-order similarity matrix to obtain a row vector and a column vector;

[0078] Weighted fusion is performed according to the row vector and the column vector to obtain a fusion feature.

[0079] Optionally, the concentration evaluation module is further used to:

[0080] Determine evaluation factors according to evaluation criteria for student behavior; the evaluation criteria include positive behavior, negative behavior, positive emotion and negative emotion; the evaluation factors include first-level evaluation factors and second-level evaluation factors; the first-level evaluation factors are a combination of behavior and emotion in the evaluation criteria; the first-level evaluation factors represent the behavior-emotion state of the student; the second-level evaluation factors refer to the behavior actions and emotion representations corresponding to the behavior-emotion state;

[0081] Setting a concentration evaluation level; assigning a value to the concentration evaluation level to obtain a concentration level value; the concentration evaluation level includes low, lower, medium, higher and high;

[0082] The analytic hierarchy process is used to assign weights to the first-level evaluation factors and the second-level evaluation privacy to obtain the evaluation factor weights;

[0083] Establish a single factor evaluation matrix based on the preset questionnaire survey results;

[0084] Based on the first-level evaluation factors and the second-level evaluation factors, according to the single-factor evaluation matrix and the evaluation factor weights, a fuzzy comprehensive evaluation is performed on the student behavior recognition results and the student emotion recognition results to obtain a concentration evaluation result.

[0085] On the other hand, a classroom concentration assessment device is provided, comprising: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the classroom concentration assessment methods that integrate behavioral analysis and sentiment analysis is implemented.

[0086] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned classroom concentration assessment methods that integrate behavioral analysis and sentiment analysis.

[0087] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0088] The present invention proposes a classroom concentration evaluation method that integrates behavior analysis and sentiment analysis. By constructing a two-stage classroom behavior recognition model, the model architecture can enrich the scenarios that the model supports recognition and improve the generalization performance of the model. Based on the comprehensive analysis of the results of behavior recognition and emotion recognition, the students' concentration is comprehensively evaluated by taking into account their behavior and emotions, and the concentration is quantified to provide a data basis for the evaluation of teaching effectiveness, help teachers adjust teaching strategies according to students' learning status, help students make self-adjustments, and improve classroom learning efficiency. The present invention is an automatic and efficient classroom concentration evaluation method that takes into account both behavior analysis and sentiment analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0090] Figure 1 This is a flow chart of a method for evaluating classroom concentration that integrates behavior analysis and sentiment analysis provided by an embodiment of the present invention;

[0091] Figure 2 It is a block diagram of a classroom concentration device integrating behavior analysis and sentiment analysis provided by an embodiment of the present invention;

[0092] Figure 3 It is a structural schematic diagram of a classroom concentration assessment device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0093] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0094] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0095] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0096] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0097] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0098] The embodiment of the present invention provides a method for evaluating classroom concentration by integrating behavior analysis and sentiment analysis. The method can be implemented by a classroom concentration evaluation device, which can be a terminal or a server. Figure 1 The flowchart of the classroom concentration assessment method integrating behavioral analysis and sentiment analysis is shown in the figure. The processing flow of the method may include the following steps:

[0099] S1. Collect data on students' classroom status through cameras to obtain training data sets; the training data sets include behavior recognition data sets and emotion recognition data sets.

[0100] Optionally, data of students' classroom status is collected through a camera to obtain a training data set, including:

[0101] Using cameras to shoot different angles of different classrooms to obtain monitoring videos; performing frame processing on the monitoring videos to obtain classroom learning pictures;

[0102] Based on the classroom learning pictures, the OpenPose algorithm is used to estimate the human body posture and obtain the coordinates of the key points of the students' bodies;

[0103] Generate images based on the coordinates of the key points of the student's body to obtain the student's RGB image and the student's skeleton image;

[0104] Based on the students' head-down, head-up and head-side behaviors, the student RGB images and student skeleton images are screened to obtain a large-scale behavior recognition dataset; the large-scale behavior recognition dataset is used to train the large-scale behavior recognition model;

[0105] Based on the students' mobile phone playing, note-taking and empty-handed behaviors, LabelImg software was used to annotate the students' RGB images and skeleton images to obtain a small-amplitude behavior recognition dataset; the small-amplitude behavior recognition dataset is used to train the small-amplitude behavior recognition model;

[0106] According to the student RGB image, the student face area image is cropped through the preset face detection algorithm to obtain the student face picture; based on the students' facial expressions of happiness, expressionlessness, fatigue, distraction and confusion, the student face pictures are screened to obtain the emotion recognition data set; the emotion recognition data set is used for the training of the emotion recognition model.

[0107] In a feasible implementation, the surveillance video shot in the classroom is framed every 20 seconds to obtain classroom learning pictures. The classroom multimodal large-scale behavior recognition dataset composed of RGB images and corresponding skeleton images has 1672, 1478 and 682 images of lowering the head, raising the head and sideways the head, respectively. The RGB images of all students are annotated with playing with mobile phones, taking notes and empty hands through LabelImg software. The annotated labels are xml files. The xml label files and the corresponding RGB images constitute a classroom small-scale behavior recognition dataset containing three behaviors of playing with mobile phones, taking notes and empty hands. The number of images of playing with mobile phones, taking notes and empty hands are 1217, 907 and 925, respectively. In the emotion recognition dataset, the number of images of pleasure, expressionlessness, fatigue, distraction and confusion are 1030, 1070, 279, 809 and 349, respectively.

[0108] S2. A two-stage classroom behavior recognition model to be trained is established based on the residual neural network structure and the Yolov8 model structure; a classroom emotion recognition model to be trained is established based on the residual neural network structure of the attention mechanism.

[0109] Among them, the two-stage classroom behavior recognition model includes a large-amplitude behavior recognition model and a small-amplitude behavior recognition model;

[0110] The large-scale behavior recognition model includes a data fusion layer, a feature fusion layer and a linear output layer.

[0111] In a feasible implementation, the two-stage classroom behavior recognition model is mainly composed of a multimodal large-amplitude behavior recognition model and a small-amplitude behavior recognition model. The large-amplitude behavior recognition model includes data layer fusion, feature layer fusion and linear layer output, and the small-amplitude behavior recognition model is a yolov8 network structure. The multimodal large-amplitude behavior recognition model is used to classify lowering the head, tilting the head and raising the head, and the small-amplitude behavior recognition model is used to classify the behaviors of playing with mobile phones and taking notes.

[0112] S3. Use the behavior recognition data set to train the two-stage classroom behavior recognition model to obtain the two-stage classroom behavior recognition model.

[0113] In a feasible implementation, a large-scale behavior recognition dataset is used to train the two-stage classroom behavior recognition model using conventional training methods.

[0114] The small-amplitude behavior recognition model is trained using the small-amplitude behavior recognition dataset used in class. A target detection model based on a single neural network (You Only Look Once V8, Yolov8) is trained using a combined loss function, which includes positioning loss, confidence loss, and classification loss.

[0115] The localization loss is used to measure the accuracy of the target bounding box, the confidence loss is used to measure the probability of the target existence, and the classification loss is used to measure the accuracy of the target category.

[0116] S4. Use the emotion recognition data set to train the classroom emotion recognition model to obtain an emotion recognition model.

[0117] In a feasible implementation, the classroom emotion recognition dataset is divided into a training set and a test set in a ratio of 7:3. The training dataset is expanded using image processing methods, including image rotation, flipping, blurring, adding noise, and changing brightness. The number of pictures of pleasure, distraction, doubt, expressionlessness, and fatigue in the expanded training dataset are 1211, 1296, 1320, 1211, and 1121, respectively.

[0118] The CosineWarmup strategy is adopted, and the emotion recognition dataset is used to train the Residual Network 50 (resnet50) model with the introduction of the attention mechanism, and the model parameters are updated through the mixup loss function.

[0119] The resnet50 model with attention mechanism is input as face image I face ∈RH×W×3 , the feature map Featrue is obtained through the convolution part and pooling of resnet50 face ∈R H×W×C , and then use the flatten layer to transform Featrue face ∈R H×W×C Expand into a one-dimensional vector Featrue face ∈R N , where N is H×W×C; the flattened feature vector is calculated through a fully connected layer and a sigmoid function to obtain the attention weight W, and the attention weight W is multiplied by the new fully connected layer output to obtain the final output result.

[0120] The mixup loss function first randomly extracts two samples x from the training sample i and x j Perform a simple random weighted sum, and the sample label y i and j It also corresponds to the weighted summation, generating new samples and tags As shown in the following formulas (1) and (2):

[0121]

[0122]

[0123] Among them, λ is a random number between 0 and 1, indicating that x i and x j The weights in the new samples.

[0124] The loss is calculated by combining the predicted result with the weighted sum of the labels. The loss function is expressed as follows (3):

[0125]

[0126] in, Represents the model's mixed input The predicted output of y; CE(·) represents the cross entropy loss function; i and j is the original sample x i and x j The corresponding true label; λ is the mixing coefficient, which controls the degree of mixing between samples. After training, the optimal model is saved as a .pth file, and then the test set is tested to evaluate the performance of the model.

[0127] S5. Obtain the current student classroom status video; perform data processing on the current student classroom status video to obtain a current status data set; input the current status data set into a two-stage classroom behavior recognition model to obtain a student behavior recognition result; input the current status data into an emotion recognition model to obtain a student emotion recognition result.

[0128] Optionally, the two-stage classroom behavior recognition model includes a large-amplitude behavior recognition model, and the current state data set is input into the two-stage classroom behavior recognition model to obtain a student behavior recognition result, including:

[0129] Divide the current state data set to obtain the current large-scale behavior recognition data set;

[0130] Input the current large-scale behavior recognition data set into the data fusion layer to obtain the fused RGB image and the fused skeleton image;

[0131] Input the fused RGB image and the fused skeleton image into the feature fusion layer to obtain the fused feature;

[0132] The fused features are input into the linear output layer for data processing to obtain large-scale behavior recognition results.

[0133] In a feasible implementation, the two-stage classroom behavior recognition model recognizes the large-scale and small-scale behaviors of students in the classroom.

[0134] Optionally, the current large-scale behavior recognition data set is input into a data fusion layer to obtain a fused RGB image and a fused skeleton image, including:

[0135] Preprocess the current large-scale behavior recognition data set to obtain a first RGB image and a first skeleton image;

[0136] Resize the first RGB image and the first skeleton image to obtain a second RGB image and a second skeleton image;

[0137] The second skeleton image is integrated into the second RGB image by using a weighted fusion method to obtain a fused RGB image;

[0138] The second RGB image is integrated into the second skeleton image by using a weighted fusion method to obtain a fused skeleton image.

[0139] In a feasible implementation, in the data fusion layer of the large-scale behavior recognition model, the input image data is preprocessed to obtain a first RGB image and a first skeleton image. The first RGB image and the first skeleton image are both adjusted to a size of 224×224 and standardized to obtain a second RGB image and a second skeleton image. The image fusion is realized by using the weighted average method, and then the weight is adjusted to control the fusion effect. The RGB image with fused skeleton information and the skeleton image with fused RGB image information can be expressed as follows. The mathematical expressions of the process are as follows (4) and (5):

[0140] F rgb =α·RGB+(1-α)·SK (4)

[0141] F sk =α·SK+(1-α)·RGB (5)

[0142] Among them, RGB represents the RGB image after image preprocessing; SK represents the skeleton image after image preprocessing; α represents the weight, and the weighted average is performed according to the weight α and (1-α) to generate the fused RGB image and skeleton image.

[0143] Optionally, the fused RGB image and the fused skeleton image are input into a feature fusion layer to obtain fused features, including:

[0144] Input the fused RGB image and the fused skeleton image into the feature fusion layer for feature extraction to obtain a first RGB feature image and a first skeleton feature image;

[0145] Resize the first RGB feature map and the first skeleton feature map to obtain a second RGB feature map and a second skeleton feature map;

[0146] Calculate according to the second RGB image and the second skeleton feature image to obtain a low-order similarity matrix;

[0147] Calculate according to the low-order similarity matrix to obtain a high-order similarity matrix;

[0148] Calculate according to the high-order similarity matrix to obtain row vectors and column vectors;

[0149] Weighted fusion is performed according to row vectors and column vectors to obtain fusion features.

[0150] In a feasible implementation, in the feature layer fusion layer of the large-scale behavior recognition model, the fused RGB image F rgb ∈R H×W×3 and skeleton graph F sk ∈R H×W×3The features are extracted through the convolution part of the residual neural network 18 (resnet18) structure to obtain F rgb Feature map Featrue after feature extraction rgb ∈R H×W×C and F sk Feature map Featrue after feature extraction sk ∈R H×W×C To facilitate subsequent calculations, we adjust the size of the feature map to obtain a feature map Featrue with a feature map size of N×C rgb ∈R N×C and Featrue sk ∈R N×C , where N is H×W.

[0151] Calculate Featrue rgb ∈R N×C and Featrue sk ∈R N×C The low-order similarity matrix LS∈R C×C , the calculation formula is as follows (6):

[0152] LS=Featrue rgb T Featrue sk (6)

[0153] Among them, the low-order similarity matrix LS∈R C×C The value of the i-th row and j-th column represents the Featrue rgb ∈R N×C The i-th channel and Featrue sk ∈R N×C The similarity of the jth channel.

[0154] By calculating the low-order similarity matrix LS∈R C×C The similarity between the i-th row and the j-th row (1≤i, j≤N) of different channels is obtained by the high-order similarity matrix HS∈R C×C , the high-order similarity matrix HS∈R C×C The calculation is done through matrix multiplication.

[0155] For the high-order similarity matrix HS∈R C×C The sum of the rows and columns of get two column vectors Sum row ∈R C×1 and Sum col ∈R C×1 , perform maximum and minimum normalization on these two vectors. The mathematical expressions of the normalization process are as follows:

[0156]

[0157]

[0158] Among them, Sum row * ∈R C×1 and Sum col * ∈R C×1 is a normalized vector; and is the maximum of two column vectors; and is the minimum of two column vectors.

[0159] The two normalized column vectors are calculated using the sigmoid activation function to obtain the weight α∈R for multimodal feature fusion C×1 and β∈R C×1 , the calculation process is as follows (9) and (10):

[0160] α=1-sigmoid(Sum row * ) (9)

[0161] β=1-sigmoid(Sum col * ) (10)

[0162] The features extracted from the RGB image and the skeleton image are fused to obtain the fused feature Featrue fusion ∈R H×W×C , the mathematical expression of the fusion process is as follows (11):

[0163] Featrue fusion =α·Featrue rgb +β·Featrue sk (11)

[0164] The fused feature Featrue fusion ∈R H×W×C After average pooling and linear layers, the large-scale behavior results are output.

[0165] S6. According to the student behavior recognition results and the student emotion recognition results, the concentration is quantified through the fuzzy comprehensive evaluation method based on the hierarchical analysis method to obtain the concentration evaluation results.

[0166] Optionally, according to the student behavior recognition results and the student emotion recognition results, the concentration is quantified by a fuzzy comprehensive evaluation method based on the hierarchical analysis method to obtain a concentration evaluation result, including:

[0167] Determine the evaluation factors according to the evaluation criteria of student behavior; the evaluation criteria include positive behavior, negative behavior, positive emotion and negative emotion; the evaluation factors include first-level evaluation factors and second-level evaluation factors; the first-level evaluation factors are the combination of behavior and emotion in the evaluation criteria; the first-level evaluation factors represent the behavior-emotion state of students; the second-level evaluation factors refer to the behavior actions and emotion representations corresponding to the behavior-emotion state;

[0168] Setting a concentration evaluation level; assigning a value to the concentration evaluation level to obtain a concentration level value; the concentration evaluation levels include low, lower, medium, higher and high;

[0169] The analytic hierarchy process is used to assign weights to the first-level evaluation factors and the second-level evaluation privacy to obtain the evaluation factor weights;

[0170] Establish a single factor evaluation matrix based on the preset questionnaire survey results;

[0171] Based on the first-level evaluation factors and the second-level evaluation factors, according to the single-factor evaluation matrix and the evaluation factor weights, a fuzzy comprehensive evaluation is performed on the student behavior recognition results and the student emotion recognition results to obtain the concentration evaluation results.

[0172] In a feasible implementation, there are eight different combinations of first-level evaluation factors: positive emotions and positive behaviors, positive emotions and negative behaviors, neutral emotions and positive behaviors, neutral emotions and negative behaviors, negative emotions and positive behaviors, negative emotions and negative behaviors, positive behaviors, and negative behaviors. Among them, positive behaviors are looking up and taking notes, negative behaviors are lowering the head, tilting the head, and playing with mobile phones, positive emotions are pleasure and confusion, neutral emotions are expressionless, and negative emotions are distraction and fatigue. The second-level evaluation factors are the behaviors and emotions corresponding to a certain combination.

[0173] The concentration evaluation levels are divided into five categories: low, lower, medium, higher and high, and the values ​​are assigned as 60, 70, 80, 90 and 100 respectively.

[0174] The weight of the first-level evaluation factor ω_1 and the weight of the second-level evaluation factor ω_2 are determined by the hierarchical analysis method. The calculation process of the weights of the first-level and second-level evaluation factors is the same:

[0175] Compare the priorities of all n factors in pairs, and quantify the importance of each factor from 1 to 9. 1 means that the two factors are equally important, 3, 5, 7, and 9 respectively indicate that one factor is slightly more important, obviously more important, strongly more important, and extremely more important than the other factor, and 2, 4, 6, and 8 represent intermediate values. After comparing the priorities, an nxn judgment matrix will be obtained. Calculate the initial weight coefficient, normalize the judgment matrix, and use the square root method to calculate the eigenvector and the maximum eigenroot. Finally, perform a consistency test on the matrix and calculate the consistency index and test coefficient.

[0176] By distributing questionnaires and conducting statistical analysis on the results of the questionnaires, the single factor evaluation matrix R of the second-level evaluation factors is obtained.

[0177] The fuzzy evaluation model of concentration is: B=W*R. After two-level evaluation, the row vector D is obtained. The concentration score, that is, the concentration evaluation result, is calculated according to the formula: Score=D*S; S is the column vector composed of concentration scores of different evaluation levels.

[0178] The present invention proposes a classroom concentration evaluation method that integrates behavior analysis and sentiment analysis. By constructing a two-stage classroom behavior recognition model, the model architecture can enrich the scenarios that the model supports recognition and improve the generalization performance of the model. Based on the comprehensive analysis of the results of behavior recognition and emotion recognition, the students' concentration is comprehensively evaluated by taking into account their behavior and emotions, and the concentration is quantified to provide a data basis for the evaluation of teaching effectiveness, help teachers adjust teaching strategies according to students' learning status, help students make self-adjustments, and improve classroom learning efficiency. The present invention is an automatic and efficient classroom concentration evaluation method that takes into account both behavior analysis and sentiment analysis.

[0179] Figure 2 The present invention is a block diagram of a classroom concentration device integrating behavior analysis and sentiment analysis according to an exemplary embodiment. The device is used in a classroom concentration evaluation method integrating behavior analysis and sentiment analysis. Figure 2 The device includes a training data set acquisition module 210, a model construction module 220, a behavior recognition model training module 230, an emotion recognition model training module 240, a student state recognition module 250 and a concentration evaluation module 260. Among them:

[0180] The training data set acquisition module 210 is used to collect data on the student's classroom status through a camera to obtain a training data set; the training data set includes a behavior recognition data set and an emotion recognition data set;

[0181] The model building module 220 is used to establish a two-stage classroom behavior recognition model to be trained based on the residual neural network structure and the Yo l ov8 model structure; and to establish a classroom emotion recognition model to be trained based on the residual neural network structure of the attention mechanism;

[0182] The behavior recognition model training module 230 is used to use the behavior recognition data set to train the two-stage classroom behavior recognition model to be trained to obtain the two-stage classroom behavior recognition model;

[0183] The emotion recognition model training module 240 is used to use the emotion recognition data set to train the classroom emotion recognition model to be trained to obtain the emotion recognition model;

[0184] The student status recognition module 250 is used to obtain the current student classroom status video; perform data processing on the current student classroom status video to obtain a current status data set; input the current status data set into a two-stage classroom behavior recognition model to obtain a student behavior recognition result; input the current status data into an emotion recognition model to obtain a student emotion recognition result;

[0185] The concentration evaluation module 260 is used to quantify the concentration according to the student behavior recognition results and the student emotion recognition results through a fuzzy comprehensive evaluation method based on the hierarchical analysis method to obtain the concentration evaluation result.

[0186] Among them, the two-stage classroom behavior recognition model includes a large-amplitude behavior recognition model and a small-amplitude behavior recognition model;

[0187] The large-scale behavior recognition model includes a data fusion layer, a feature fusion layer and a linear output layer.

[0188] Optionally, the training data set acquisition module 210 is further used to:

[0189] Using cameras to shoot different angles of different classrooms to obtain monitoring videos; performing frame processing on the monitoring videos to obtain classroom learning pictures;

[0190] Based on the classroom learning pictures, the OpenPose algorithm is used to estimate the human body posture and obtain the coordinates of the key points of the students' bodies;

[0191] Generate images based on the coordinates of the key points of the student's body to obtain the student's RGB image and the student's skeleton image;

[0192] Based on the students' head-down, head-up and head-side behaviors, the student RGB images and student skeleton images are screened to obtain a large-scale behavior recognition dataset; the large-scale behavior recognition dataset is used to train the large-scale behavior recognition model;

[0193] Based on the students' mobile phone playing, note-taking and empty-handed behaviors, Label Img software was used to annotate the students' RGB images and skeleton images to obtain a small-amplitude behavior recognition dataset; the small-amplitude behavior recognition dataset is used to train the small-amplitude behavior recognition model;

[0194] According to the student RGB image, the student face area image is cropped through the preset face detection algorithm to obtain the student face picture; based on the students' facial expressions of happiness, expressionlessness, fatigue, distraction and confusion, the student face pictures are screened to obtain the emotion recognition data set; the emotion recognition data set is used for the training of the emotion recognition model.

[0195] Optionally, the student status identification module 250 is further used to:

[0196] Divide the current state data set to obtain the current large-scale behavior recognition data set;

[0197] Input the current large-scale behavior recognition data set into the data fusion layer to obtain the fused RGB image and the fused skeleton image;

[0198] Input the fused RGB image and the fused skeleton image into the feature fusion layer to obtain the fused feature;

[0199] The fused features are input into the linear output layer for data processing to obtain large-scale behavior recognition results.

[0200] Optionally, the student status identification module 250 is further used to:

[0201] Preprocess the current large-scale behavior recognition data set to obtain a first RGB image and a first skeleton image;

[0202] Resize the first RGB image and the first skeleton image to obtain a second RGB image and a second skeleton image;

[0203] The second skeleton image is integrated into the second RGB image by using a weighted fusion method to obtain a fused RGB image;

[0204] The second RGB image is integrated into the second skeleton image by using a weighted fusion method to obtain a fused skeleton image.

[0205] Optionally, the student status identification module 250 is further used to:

[0206] Input the fused RGB image and the fused skeleton image into the feature fusion layer for feature extraction to obtain a first RGB feature image and a first skeleton feature image;

[0207] Resize the first RGB feature map and the first skeleton feature map to obtain a second RGB feature map and a second skeleton feature map;

[0208] Calculate according to the second RGB image and the second skeleton feature image to obtain a low-order similarity matrix;

[0209] Calculate according to the low-order similarity matrix to obtain a high-order similarity matrix;

[0210] Calculate according to the high-order similarity matrix to obtain row vectors and column vectors;

[0211] Weighted fusion is performed according to row vectors and column vectors to obtain fusion features.

[0212] Optionally, the concentration evaluation module 260 is further used to:

[0213] Determine the evaluation factors according to the evaluation criteria of student behavior; the evaluation criteria include positive behavior, negative behavior, positive emotion and negative emotion; the evaluation factors include first-level evaluation factors and second-level evaluation factors; the first-level evaluation factors are the combination of behavior and emotion in the evaluation criteria; the first-level evaluation factors represent the behavior-emotion state of students; the second-level evaluation factors refer to the behavior actions and emotion representations corresponding to the behavior-emotion state;

[0214] Setting a concentration evaluation level; assigning a value to the concentration evaluation level to obtain a concentration level value; the concentration evaluation levels include low, lower, medium, higher and high;

[0215] The analytic hierarchy process is used to assign weights to the first-level evaluation factors and the second-level evaluation privacy to obtain the evaluation factor weights;

[0216] Establish a single factor evaluation matrix based on the preset questionnaire survey results;

[0217] Based on the first-level evaluation factors and the second-level evaluation factors, according to the single-factor evaluation matrix and the evaluation factor weights, a fuzzy comprehensive evaluation is performed on the student behavior recognition results and the student emotion recognition results to obtain the concentration evaluation results.

[0218] The present invention proposes a classroom concentration evaluation method that integrates behavior analysis and sentiment analysis. By constructing a two-stage classroom behavior recognition model, the model architecture can enrich the scenarios that the model supports recognition and improve the generalization performance of the model. Based on the comprehensive analysis of the results of behavior recognition and emotion recognition, the students' concentration is comprehensively evaluated by taking into account their behavior and emotions, and the concentration is quantified to provide a data basis for the evaluation of teaching effectiveness, help teachers adjust teaching strategies according to students' learning status, help students make self-adjustments, and improve classroom learning efficiency. The present invention is an automatic and efficient classroom concentration evaluation method that takes into account both behavior analysis and sentiment analysis.

[0219] Figure 3is a structural diagram of a classroom concentration assessment device provided by an embodiment of the present invention, such as Figure 3 As shown, the classroom concentration assessment device may include the above Figure 2 The classroom concentration device integrating behavior analysis and sentiment analysis is shown. Optionally, the classroom concentration evaluation device 310 may include a first processor 2001 .

[0220] Optionally, the classroom concentration assessment device 310 may further include a memory 2002 and a transceiver 2003 .

[0221] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0222] Combine the following Figure 3 The components of the classroom concentration assessment device 310 are specifically introduced:

[0223] The first processor 2001 is the control center of the classroom concentration assessment device 310, which can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0224] Optionally, the first processor 2001 can perform various functions of the classroom concentration assessment device 310 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0225] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 3 CPU0 and CPU1 are shown in FIG.

[0226] In a specific implementation, as an embodiment, the classroom concentration assessment device 310 may also include multiple processors, such as Figure 3The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0227] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0228] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0229] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0230] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 3 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0231] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and communicate with the first processor 2001 through the interface circuit ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0232] It should be noted that Figure 3 The structure of the classroom concentration assessment device 310 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0233] In addition, the technical effects of the classroom concentration assessment device 310 can refer to the technical effects of the classroom concentration assessment method that integrates behavior analysis and sentiment analysis described in the above method embodiment, and will not be repeated here.

[0234] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0235] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0236] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0237] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0238] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0239] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0240] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0241] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0242] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0243] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0244] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0245] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0246] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A classroom concentration assessment method integrating behavioral analysis and sentiment analysis, characterized in that: The method comprises: Collect data on students' classroom status through cameras to obtain a training data set; the training data set includes a behavior recognition data set and an emotion recognition data set; A two-stage classroom behavior recognition model to be trained is established based on the residual neural network structure and the Yolov8 model structure; a classroom emotion recognition model to be trained is established based on the residual neural network structure of the attention mechanism; Using the behavior recognition data set, training the two-stage classroom behavior recognition model to be trained to obtain a two-stage classroom behavior recognition model; Wherein, the two-stage classroom behavior recognition model includes a large-amplitude behavior recognition model and a small-amplitude behavior recognition model; The large-amplitude behavior recognition model includes a data fusion layer, a feature fusion layer and a linear output layer; The training process of the small-amplitude behavior recognition model adopts a combined loss function for training; the combined loss function includes a positioning loss function, a confidence loss function and a classification loss function; the positioning loss function is used to measure the accuracy of the target bounding box; the confidence loss function is used to measure the probability of the existence of small-amplitude behaviors; the classification loss function is used to measure the accuracy of small-amplitude behavior classification; Using the emotion recognition data set, training the classroom emotion recognition model to be trained to obtain an emotion recognition model; Acquire a current student classroom status video; perform data processing on the current student classroom status video to obtain a current status data set; input the current status data set into the two-stage classroom behavior recognition model to obtain a student behavior recognition result; input the current status data into the emotion recognition model to obtain a student emotion recognition result; The two-stage classroom behavior recognition model includes a large-scale behavior recognition model, and the inputting of the current state data set into the two-stage classroom behavior recognition model to obtain the student behavior recognition result includes: Dividing the current state data set to obtain a current large-scale behavior recognition data set; Inputting the current large-scale behavior recognition data set into the data fusion layer to obtain a fused RGB image and a fused skeleton image; Inputting the fused RGB image and the fused skeleton image into the feature fusion layer to obtain fused features; Inputting the fused features into the linear output layer for data processing to obtain a large-scale behavior recognition result; The step of inputting the fused RGB image and the fused skeleton image into the feature fusion layer to obtain fused features includes: Inputting the fused RGB image and the fused skeleton image into the feature fusion layer for feature extraction to obtain a first RGB feature image and a first skeleton feature image; Resize the first RGB feature map and the first skeleton feature map to obtain a second RGB feature map and a second skeleton feature map; Calculating according to the second RGB feature map and the second skeleton feature map to obtain a low-order similarity matrix; Calculating according to the low-order similarity matrix to obtain a high-order similarity matrix; Calculate according to the high-order similarity matrix to obtain a row vector and a column vector; Perform weighted fusion according to the row vector and the column vector to obtain a fusion feature; According to the student behavior recognition results and the student emotion recognition results, concentration is quantified by a fuzzy comprehensive evaluation method based on the hierarchical analysis method to obtain a concentration evaluation result.

2. The classroom concentration assessment method integrating behavior analysis and sentiment analysis according to claim 1 is characterized in that: The method of collecting data on the student's classroom status through a camera to obtain a training data set includes: Using cameras to shoot different angles of different classrooms to obtain monitoring videos; performing frame processing on the monitoring videos to obtain classroom learning pictures; Based on the classroom learning pictures, the OpenPose algorithm is used to estimate the human body posture and obtain the coordinates of the key points of the students' bodies; Generate a picture according to the coordinates of the key points of the student's body to obtain a student RGB picture and a student skeleton picture; Based on the student's head-lowering, head-raising and head-sideways behavior movements, the student RGB image and the student skeleton image are screened to obtain a large-amplitude behavior recognition data set; the large-amplitude behavior recognition data set is used for training a large-amplitude behavior recognition model; Based on the students' mobile phone playing, note-taking and empty-handed behaviors, the student RGB image and the student skeleton image are labeled using Label Img software to obtain a small amplitude behavior recognition data set; the small amplitude behavior recognition data set is used for the training of the small amplitude behavior recognition model; According to the student RGB image, the student face area image is cropped by a preset face detection algorithm to obtain a student face picture; based on the students' facial expressions of joy, expressionlessness, fatigue, distraction and confusion, the student face picture is screened to obtain an emotion recognition data set; the emotion recognition data set is used for training the emotion recognition model.

3. The classroom concentration assessment method integrating behavior analysis and sentiment analysis according to claim 1 is characterized in that: The step of inputting the current large-scale behavior recognition data set into the data fusion layer to obtain a fused RGB image and a fused skeleton image includes: Preprocessing the current large-scale behavior recognition data set to obtain a first RGB image and a first skeleton image; Resize the first RGB image and the first skeleton image to obtain a second RGB image and a second skeleton image; Using a weighted fusion method, the second skeleton image is integrated into the second RGB image to obtain a fused RGB image; The second RGB image is integrated into the second skeleton image by using a weighted fusion method to obtain a fused skeleton image.

4. The classroom concentration assessment method integrating behavior analysis and sentiment analysis according to claim 1 is characterized in that: The method of quantifying the concentration level by a fuzzy comprehensive evaluation method based on the analytic hierarchy process according to the student behavior recognition result and the student emotion recognition result to obtain the concentration evaluation result includes: Determine evaluation factors according to evaluation criteria for student behavior; the evaluation criteria include positive behavior, negative behavior, positive emotion and negative emotion; the evaluation factors include first-level evaluation factors and second-level evaluation factors; the first-level evaluation factors are a combination of behavior and emotion in the evaluation criteria; the first-level evaluation factors represent the behavior-emotion state of the student; the second-level evaluation factors refer to the behavior actions and emotion representations corresponding to the behavior-emotion state; Setting a concentration evaluation level; assigning a value to the concentration evaluation level to obtain a concentration level value; the concentration evaluation level includes low, lower, medium, higher and high; The analytic hierarchy process is used to assign weights to the first-level evaluation factors and the second-level evaluation factors to obtain the evaluation factor weights; Establish a single factor evaluation matrix based on the preset questionnaire survey results; Based on the first-level evaluation factors and the second-level evaluation factors, according to the single-factor evaluation matrix and the evaluation factor weights, a fuzzy comprehensive evaluation is performed on the student behavior recognition results and the student emotion recognition results to obtain a concentration evaluation result.

5. A classroom concentration device integrating behavior analysis and sentiment analysis, wherein the classroom concentration device integrating behavior analysis and sentiment analysis is used to implement the classroom concentration evaluation method integrating behavior analysis and sentiment analysis as described in any one of claims 1 to 4, characterized in that: The device comprises: A training data set acquisition module is used to collect data on the student's classroom status through a camera to obtain a training data set; the training data set includes a behavior recognition data set and an emotion recognition data set; Model building module, used to establish a two-stage classroom behavior recognition model to be trained based on the residual neural network structure and the Yolov8 model structure; to establish a classroom emotion recognition model to be trained based on the residual neural network structure of the attention mechanism; A behavior recognition model training module is used to use the behavior recognition data set to train the two-stage classroom behavior recognition model to be trained to obtain a two-stage classroom behavior recognition model; Wherein, the two-stage classroom behavior recognition model includes a large-amplitude behavior recognition model and a small-amplitude behavior recognition model; The large-amplitude behavior recognition model includes a data fusion layer, a feature fusion layer and a linear output layer; The training process of the small-amplitude behavior recognition model adopts a combined loss function for training; the combined loss function includes a positioning loss function, a confidence loss function and a classification loss function; the positioning loss function is used to measure the accuracy of the target bounding box; the confidence loss function is used to measure the probability of the existence of small-amplitude behaviors; the classification loss function is used to measure the accuracy of small-amplitude behavior classification; An emotion recognition model training module is used to use the emotion recognition data set to train the classroom emotion recognition model to be trained to obtain an emotion recognition model; The student status recognition module is used to obtain the current student classroom status video; perform data processing on the current student classroom status video to obtain a current status data set; input the current status data set into the two-stage classroom behavior recognition model to obtain a student behavior recognition result; input the current status data into the emotion recognition model to obtain a student emotion recognition result; The two-stage classroom behavior recognition model includes a large-scale behavior recognition model, and the inputting of the current state data set into the two-stage classroom behavior recognition model to obtain the student behavior recognition result includes: Dividing the current state data set to obtain a current large-scale behavior recognition data set; Inputting the current large-scale behavior recognition data set into the data fusion layer to obtain a fused RGB image and a fused skeleton image; Inputting the fused RGB image and the fused skeleton image into the feature fusion layer to obtain fused features; Inputting the fused features into the linear output layer for data processing to obtain a large-scale behavior recognition result; The step of inputting the fused RGB image and the fused skeleton image into the feature fusion layer to obtain fused features includes: Inputting the fused RGB image and the fused skeleton image into the feature fusion layer for feature extraction to obtain a first RGB feature image and a first skeleton feature image; Resize the first RGB feature map and the first skeleton feature map to obtain a second RGB feature map and a second skeleton feature map; Calculating according to the second RGB feature map and the second skeleton feature map to obtain a low-order similarity matrix; Calculating according to the low-order similarity matrix to obtain a high-order similarity matrix; Calculate according to the high-order similarity matrix to obtain a row vector and a column vector; Perform weighted fusion according to the row vector and the column vector to obtain a fusion feature; The concentration evaluation module is used to quantify the concentration according to the student behavior recognition result and the student emotion recognition result through a fuzzy comprehensive evaluation method based on the hierarchical analysis method to obtain the concentration evaluation result.

6. A classroom concentration assessment device, characterized in that: The classroom concentration assessment device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 4.

Images