A student classroom abnormal behavior detection method and detection system based on multi-modal data fusion

Abstract

The application discloses a student classroom abnormal behavior detection method based on multi-modal data fusion, and the method comprises the following steps: pre-processing student expression data and body data in a classroom video and selecting an interesting area, using an LBP histogram as a classroom student expression feature description, adopting a single-class support vector machine (One-Class SVM) as a classifier to perform expression abnormality detection, using a flow histogram method (HOFO) to calculate a body feature vector in the classroom video stream, using a single support vector machine (One-Class SVM) as a classifier to perform body action abnormality recognition, and finally performing logical decision fusion to obtain a final classroom abnormal behavior detection result. The application adopts a new multi-modal data fusion method, can comprehensively integrate data information of multiple modes, and makes the student classroom behavior detection result more accurate. The application further discloses a detection system for realizing the above method.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of educational informatization, and particularly relates to a student classroom abnormal behavior detection method and system based on multi-modal data fusion. BACKGROUND

[0002] As the long of the crowd, humans can use visual system, auditory system, olfactory system and tactile system and other sensory systems to accept information from the external environment, form a "mirror image" of the real world in the brain through multi-modal information synthesis. The biological information processing system can suppress environmental noise, extract key information from each sensory system, solve information conflict and make decisions. Intelligent machines can also receive multi-modal information from the external environment through multiple information channels, such as video information, audio information, etc., but there is still a big gap between artificial intelligence and humans in multi-modal data processing capability. There are still many problems to be solved in multi-modal data fusion, such as suppressing different noises in different modal data, information conflict between multi-modal data, and how to fuse multi-modal data to improve the accuracy of final recognition or decision. Therefore, fusing multi-modal data helps to solve ambiguity and improve the accuracy of intelligent decision-making.

[0003] In the field of education, Multimodal Learning Analytics (MMLA) focuses on collecting and analyzing various traces obtained from different aspects of the learning process to better understand and improve the teaching process. The method of multi-modal data fusion can be roughly divided into three categories according to the level of information fusion, including pixel-level fusion, feature-level fusion, decision-level fusion and combination of various methods. Each modality of multi-modal data contains more or less information that helps the final classification or recognition task. In order to more effectively fuse multi-modal data, it is necessary to understand the level of information fusion and apply different information fusion strategies for different situations.

[0004] In the field of education, the classroom is the main place for education practice, and scientific and effective classroom observation can help students improve learning efficiency and help teachers improve teaching methods and strategies. Student classroom behavior is an important indicator of classroom participation, and body movements and facial expressions can convey 70% of overall learning emotional information, which is an indispensable non-verbal behavior for measuring student classroom participation. Looking at the current research on student classroom behavior detection supported by video recognition technology in the field of education, it is found that most research focuses on single-modal data. For example, He Xiuling et al. [1] proposed a student classroom behavior recognition method based on human skeleton and deep learning, that is, extracting human skeleton single-modal information from images and combining CNN-10 to recognize student classroom behavior. Similarly, Xu Jiazhen et al. [2]Taking the actual classroom teaching video shot by a monitoring device as a data source, student single modal human skeleton information is proposed, and Boosting algorithm and convolutional neural network are used for automatic identification. [3] Applying the limb motion recognition technology to the remote teaching system helps to feedback the learning state of the remote learners. [4] Based on the classroom video data, the student state analysis is performed on the number of students in the classroom, the student position and the student face key points, and the classroom activity level is calculated by statistical calculation. In summary, the student state detection in the classroom scene still faces the following challenges: first, there is still a lack of detection of student classroom state abnormal events; second, the student classroom state detection based on video data focuses on single modal data and does not fully integrate multi-modal information; third, in terms of method, the feature descriptor is mostly dependent on manually made features or unsupervised methods of stacking denoising autoencoders. SUMMARY

[0005] In order to solve the problems existing in the prior art, the purpose of the present application is to provide a student classroom abnormal state detection method and system based on multi-modal data fusion. The method uses local binary pattern (LBP) and optical flow histogram (HOFO) methods to form expression feature vectors and limb feature vectors for two different modal data contained in the classroom video stream. After dimension reduction of the feature vectors, they are respectively input into one-class support vector machine (One-class SVM) to obtain abnormal detection results of different modalities. Finally, the final fusion abnormal detection result is obtained through a decision fusion method, so as to achieve the effect of fusing multi-modal information and improving the detection accuracy. The abnormal state includes expression abnormality and limb abnormality; wherein the expression abnormality includes crying, disgust, dull eyes, ghost expression, etc.; the limb abnormality includes eating, playing mobile phone, pushing with hand, punching, standing, waving hand, yawning, etc.

[0006] The present application provides a student classroom abnormal behavior detection method based on multi-modal data fusion, which comprises the following steps:

[0007] Step one, capturing video stream data of student classroom learning through a classroom camera;

[0008] Step two, preprocessing the classroom learning video stream data in step one;

[0009] The preprocessing includes adjusting the original video image resolution to 256*1169 to ensure image clarity, and performing image normalization to adjust different video images to the same size of pixel value, and performing image graying and noise filtering on the basis of the above operations;

[0010] Among them, the color image matrix captured by the camera requires a large storage space, which is not suitable for image processing. Grayscale processing refers to replacing the R, G, and B component values ​​of the color image with a single value to represent the color depth of the image pixel. The value range is [0, 255]. This invention adopts the weighted average method; that is, the weighted average of the R, G, and B component values ​​of the pixel in the color image is used as the grayscale value of the grayscale image. The weight values ​​of the pixel in the R, G, and B components are selected according to the actual situation.

[0011] Noise is present from time to time in the image data of the captured video stream. Noise refers to unnecessary or redundant information in the image data, which is an important factor that interferes with image quality. Gaussian filtering is used to remove noise.

[0012] Step 3: Binarize the image after preprocessing in Step 2. Image binarization converts the image into a representation mode with only two colors: black and white. The color depth of each pixel is 0 or 255. The Ostu method (maximum inter-class difference) is used to select regions of interest in the video, which facilitates the extraction of student facial expressions and body movement features. The specific steps are as follows:

[0013] The pixels in the classroom video image are divided into background pixels and foreground pixels. Background pixels are those contained in the stable image background model, while foreground pixels, relative to the background pixels, roughly contain the foreground moving targets. Specifically, a background model is first established using the ViBe algorithm. Then, a difference image is obtained based on the difference between the current frame and the background model, and a specific threshold T is set for binarization. The threshold T is initially set using the average pixel value of the image and is adjusted according to the specific training process. The mathematical expression is as follows. If the environment changes, the background model needs to be continuously updated to adapt to the changes in the background environment. Morphological processing is performed on the binarized image to extract edges and obtain the foreground moving targets.

[0014]

[0015] Among them, f k (x,y) represents the image of the Kth frame, B k (x,y) is the currently constructed background model, and T is the threshold used for image segmentation.

[0016] Step four: Extract the facial expression feature vector and body feature vector of students in the classroom from the processed classroom video stream data using LBP feature histogram and optical flow histogram (HOFO) methods, respectively.

[0017] Local Binary Pattern (LBP) offers strong discriminative power and low computational complexity, making it suitable for appearance feature extraction. It calculates pixel values ​​by comparing the grayscale value of the image's center pixel P with the grayscale values ​​of its neighboring pixels. (See...) Figure 2 The formula is expressed as follows:

[0018] L approximates l (s (n0-n c ),...,s (n7-n c ))

[0019] Wherein, L represents the value obtained by calculation, l (x) represents the single s (x) calculation result, n0 represents the pixel value in the window except the center pixel, n c represents the center pixel value,

[0020] Figure 2 The LBP operator calculation process is shown, and 8-bit binary numbers are read in sequence as the feature values of pixels, and the binary numbers are converted into decimal numbers according to the following formula.

[0021]

[0022] Wherein, (x c ,y c ) is the position of the center pixel, n i represents the pixel value in the window except the center pixel, and the values from the top left corner of the window are taken as 0, 1, 2…7 in sequence.

[0023] The calculation steps of the LBP feature vector are as follows: first, the student's face expression window is detected, the window is divided into small cell bodies with a size of 20*15, the LBP value of each pixel in each cell is calculated, then the histogram of each cell is counted and normalized, finally, the histograms of all cells are connected to form the LBP feature vector.

[0024] Step five, the student's body movement is accompanied by changes in position and direction, and the optical flow is composed of the brightness changes of the image, which can be used to express the motion information of the target point, the present application uses the optical flow histogram (HOFO) to count the motion information of the pixels, in order to reduce the calculation amount, before that, the ViBe algorithm is used to extract the foreground pixels and determine the foreground interest area, the foreground interest area covers the activity area of the student.

[0025] The descriptor calculation in the optical flow histogram (HOFO) is in block units, including b w ×b h cell bodies (cells), each cell contains c w ×c hEach cell contains pixels, where w and h represent the length and width (including the block's length and width, and the cell's length and width), respectively. A one-dimensional histogram of all pixels is calculated within each cell. This invention uses eight bins to statistically analyze the horizontal and vertical optical flow direction information, dividing the optical flow direction 360° into eight directional blocks. Each pixel within a cell is mapped to a fixed angle range using the optical flow direction, and statistical histograms are used to obtain the optical flow histogram for that cell. For example, if the mapped angle of a pixel's optical flow direction is 0° to 45°, the first bin of the histogram is incremented by 1. Then, several cells are combined into a block, and the feature descriptors of all cells within a block are concatenated to obtain the HOFO feature descriptor for that block, such as... Figure 3 .

[0026] For example, such as Figure 4 The feature vector descriptor F of the i-th frame i The calculation process involves calculating the angle between the optical flow vector and the horizontal axis based on the bins projected onto the corresponding intervals. Each pixel in the cell is mapped to a fixed angular range according to the optical flow direction, thus generating a gradient histogram for the cell, which is then used for statistical analysis. For example, if the mapped angle of the pixel's optical flow direction falls within (0°-45°), then the first bin of the histogram (…)… Figure 5 Add 1 to the right side, then connect the cell bodies into a block, and connect the feature descriptors of all cell bodies in each block to obtain the HOFO feature descriptor of the entire block.

[0027] Before performing step six, a classifier needs to be trained using data on students' normal classroom behavior. One-Class Support Vector Machine (SVM) is a special variant of SVM that only requires training with one class of samples (e.g., a dataset of normal behavior samples).

[0028] The One-Class Support Vector Machine (OCSVM) works as follows: OCSVM aims to establish a decision boundary that maximizes the distance between the normal dataset and the origin. The optimal learning boundary is trained using a normal sample dataset, encompassing almost all normal behavior sample points. During testing, points within the boundary and close to the origin are considered normal behavior, while points outside the boundary are considered outliers. OCSVM maps the training dataset to a high-dimensional feature space using the kernel function K(x,z)=(Φ(x)·Φ(z)), improving the clustering of the input data. It then iteratively finds the optimal edge hyperplane, maximizing the distance between the training data and the origin. Assume the training dataset... To obtain the boundary conditions for a normal dataset, the model is optimized as follows:

[0029]

[0030] where w and p are parameters used to determine the decision boundary, which can be located in the optimization process. i represents the training sample set, N is the total number of training data, v is a trade-off parameter, and i ξ n ={ξ1,ξ2,...,ξ i} are the slack variables of x 2 , and Φ represents the distance from the sample set to the origin of coordinates. Φ: represents a nonlinear mapping from the data input space χ to the feature space

[0031] K(x,z) = (Φ(x) · Φ(z)) (2)

[0032] where K(x,z) and Φ(x) represent the kernel function and the mapping function, respectively. Φ(x) · Φ(z) is the inner product of Φ(x) and Φ(z). x,z∈χ, represents the sample of the data input space.

[0033] For example, the Gaussian kernel function is as follows:

[0034]

[0035] where z represents the center of the kernel function, and ||x-z|| i represents the Euclidean distance between vector x and vector z. The Gaussian kernel value decreases monotonically as the distance between the two vectors increases, and σ controls the range of the Gaussian kernel function.

[0036] The solution of formula (1) can be obtained by Lagrange Multiplier, as shown in formula (4)

[0037]

[0038] where α N is the Lagrange operator, and let α = [α1, α2,..., α T ]. w (such as the optical flow positive vector) is defined by . represents the boundary parameter that can be calculated by any x j , and then the best boundary is determined by support vector expansion. The decision function is as follows:

[0039]

[0040] where α i represents the Lagrange operator, f(x) represents the decision function, and Σ represents the summation function. x isign (x) represents a non-zero training sample, and sign() represents a sign function.

[0041] Step six, based on the LBP feature vector of step four and the optical flow histogram (HOFO) of step five, the expression abnormality classifier and the limb action abnormality classifier are trained respectively, and then the preprocessed classroom learning test data set is input, and the classification results of whether the expression and the limb action are abnormal are obtained respectively. In the present application, crying, disgust, glassy stare, ghost expression and the like belong to abnormal expressions; eating, playing mobile phones, pushing with hands, punching, standing, waving hands, yawning and the like belong to abnormal classroom behaviors.

[0042] Step seven, the two modal classification results of the expression and the limb are logically fused to obtain the final classroom abnormal behavior detection result. Before this step, the detection results of the student expression and the limb action are obtained, R e and R l respectively represent the detection results of the two, and R represents the final detection result.

[0043]

[0044]

[0045] R=R e ∩R l

[0046] wherein R represents the overall detection result. When R=1, the student behavior is normal, that is, R e =1, R l =1. When any one of the expression detection and the limb action detection is abnormal, the overall result is abnormal, that is, R=0.

[0047] The present application also provides a detection system for realizing the above method, and the system comprises a classroom learning behavior database module, an expression abnormality detection module, a limb action abnormality detection module and a decision fusion module.

[0048] The classroom learning behavior database module is used for collecting student classroom video data, determining a student classroom learning expression data set and a limb action data set; pre-processing the training data set, selecting a region of interest in a video image by borrowing a method in an OpenCV library, and saving the processed data set in a classroom learning video database and marking it with a normal or abnormal label.

[0049] The expression abnormality detection module inputs the expression to be detected into the trained model to obtain the result of whether the expression is abnormal.

[0050] The limb movement anomaly detection module obtains the result of whether the limb movement is abnormal by inputting the limb movement to be detected into the trained model.

[0051] The decision fusion module adopts a decision-level fusion strategy to logically fuse the expression anomaly detection result and the limb movement anomaly detection result, and judges the final behavior anomaly detection result.

[0052] The beneficial effects of the present application include: the present application designs a student classroom abnormal behavior detection method based on a semi-supervised method, can systematically fuse two modal information of expression and limb movement, can highlight the unique features of each modal, the information is more sufficient, has high fault tolerance, improves the accuracy of the detection result. In addition, the present application proposes to use a semi-supervised learning method One-Class SVM as a classifier, this method has less requirements for the number of samples, and has high timeliness. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 It is a student classroom abnormal behavior detection method based on multi-modal data fusion of the present application.

[0054] Figure 2 It is an LBP operator calculation diagram provided by the present application.

[0055] Figure 3 It is a histogram of optical flow (HOFO) calculation step provided by the present application.

[0056] Figure 4 It is an i-th frame HOFO calculation process provided by the present application.

[0057] Figure 5 It is a HOFO extraction visualization process of the present application.

[0058] Figure 6 It is a flow chart of a real-time detection method for student abnormal behavior in a real classroom implemented by the present application.

[0059] Figure 7 It is a functional module diagram implemented by the present application. DETAILED DESCRIPTION

[0060] The present application will be further described in detail in combination with the following specific embodiments and drawings. The process, condition, experimental method, etc. for implementing the present application are the general knowledge and common sense in the art, and the present application does not have special limitations.

[0061] The application discloses a student classroom abnormal behavior detection method based on multi-modal data fusion, and the method comprises the following steps: pre-processing student expression data and body data in a classroom video and selecting a region of interest (ROI), using an LBP histogram as a classroom student expression feature description, adopting a one-class support vector machine (One-Class SVM) as a classifier to perform expression abnormality detection, using a flow histogram method (HOFO) to calculate a body feature vector in a classroom video stream, using a one-class support vector machine (One-Class SVM) as a classifier to perform body action abnormality recognition, and finally performing logical decision fusion to obtain a final classroom abnormal behavior detection result.

[0062] Specifically,

[0063] The application discloses a student classroom abnormal behavior detection method based on multi-modal data fusion, and the method comprises the following steps: pre-processing student expression data and body data in a classroom video and selecting a region of interest (ROI), using an LBP histogram as a classroom student expression feature description, adopting a one-class support vector machine (One-Class SVM) as a classifier to perform expression abnormality detection, using a flow histogram method (HOFO) to calculate a body feature vector in a classroom video stream, using a one-class support vector machine (One-Class SVM) as a classifier to perform body action abnormality recognition, and finally performing logical decision fusion to obtain a final classroom abnormal behavior detection result.

[0064] Step 1: collecting student classroom learning data by using a camera, wherein the data set is divided into a training data set and a test data set, the training data set comprises an expression training data set and a body action training data set, and a detection database is established;

[0065] Step 2: pre-processing the expression training data set and the body action data set, including image resolution adjustment, image noise reduction, image size normalization and image graying.

[0066] Step 3: determining a region of interest containing a to-be-detected target, and then performing feature extraction and classifier training of the expression and the body action respectively.

[0067] Step 4: inputting the test set into two trained classifiers respectively to obtain an expression abnormality detection result and a body action abnormality detection result respectively.

[0068] Step 5: performing logical fusion on the detection results to obtain a final classroom behavior abnormality detection.

[0069] Embodiment:

[0070] The embodiment provides a student abnormal behavior real-time detection method for a real classroom, a flow chart is shown in Figure 6 , and a functional module diagram is shown in Figure 7 .

[0071] (1) establishing a student classroom learning video database, Figure 7 The specific content of establishing the student classroom learning video database in the step is as follows:

[0072] The student classroom learning video database is established in the classroom behavior database module, student classroom video data is collected, student classroom learning expression data set and body movement data set are determined, and the training expression OCSVM and body movement OCSVM are prepared respectively; the training data set is preprocessed, including video clarity, resolution, video size normalization and video noise reduction; the region of interest in the video image is selected by borrowing the method in the OpenCV library, the relative position and marked pixels of the region of interest are calculated, the width and length of the selected rectangular frame are input, the rectangular function is input to extract the region of interest, and all pixel values outside the region of interest are marked as 0. After the above steps, the processed data set is saved in the classroom learning video database, and is labeled as normal or abnormal.

[0073] (2) Expression anomaly detection, Figure 7 The specific content of expression anomaly detection is as follows:

[0074] The expression anomaly detection is performed by the expression anomaly detection module and is divided into two stages, an offline training stage of an expression classifier and an online test stage, wherein the student classroom expression data set includes a normal expression training set and a test set. In the training stage, the student classroom normal expression data set is selected, and video clarity, resolution, video size normalization and noise reduction preprocessing are performed to ensure the video quality of the training data set; LBP histogram is used to extract expression features to form a feature vector, which is used as an input value for OCSVM classifier training. In the test stage, the test data set is composed of student classroom learning videos, the expression features are extracted after preprocessing, and the trained OCSVM classifier is input for classification, and the abnormal expression is detected as an abnormal point by OCSVM.

[0075] (3) Body movement anomaly detection, Figure 7 The specific content of body movement anomaly detection is as follows:

[0076] The body movement anomaly detection is performed by the body movement anomaly detection module and is divided into two stages, an offline training stage of a body movement classifier and an online test stage, wherein the student classroom body movement data set includes a normal body movement training set and a test set. In the training stage, the student classroom normal body movement data set is selected, and video clarity, resolution, video size normalization and noise reduction preprocessing are performed to ensure the video quality of the training data set; optical flow histogram is used to extract body movement features to form a feature vector, which is used as an input value for OCSVM classifier training. In the test stage, the test data set is composed of student classroom learning videos, the body movement features are extracted after preprocessing, and the trained OCSVM classifier is input for classification, and the abnormal body movement is detected as an abnormal point by OCSVM.

[0077] (4) decision fusion, Figure 7 The specific content of the decision fusion is as follows:

[0078] The decision fusion is carried out through a decision fusion module, expression abnormality detection results and limb action abnormality detection results are logically fused by adopting a decision-level fusion strategy, only when the logical fusion result is 1, the behavior of the student is normal, if the logical fusion result is 0, at least one of the expression and the limb action of the student is abnormal. Finally, the classroom abnormality detection result of the student is output in real time, and a teacher is prompted.

[0079] REFERENCES

[0080] [1] He X, Yang F, Chen Z, Fang J, Li Y. Student classroom behavior recognition based on human skeleton and deep learning [J]. Modern Education Technology, 2020, 30(11): 105-112.

[0081] [2] Xu J, Deng W, Wei Y. Automatic recognition of student classroom behavior based on human skeleton information extraction [J]. Modern Education Technology, 2020, 30(05): 108-113.

[0082] [3] Vermun K, Senapaty M, Sankhla A, et al. Gesture-based affective and cognitive states recognition using kinect for effective feedback during elearning [A]. 2013 IEEE Fifth International Conference on Technology for Education (t4e 2013) [C]. Piscataway: IEEE, 2013: 107-110.

[0083] [4] Jia L, Zhang Z, Zhao X, Yan X. Classroom student state analysis based on artificial intelligence video processing [J]. Modern Education Technology, 2019, 29(12): 82-88.

[0084] The protection scope of the present application is not limited to the above embodiments. Changes and advantages that can be thought of by those skilled in the art without departing from the spirit and scope of the present application are included in the present application, and are protected by the appended claims.

Claims

1. A student classroom abnormal behavior detection method based on multi-modal data fusion, characterized in that, The method comprises: Step one, obtaining video data of students in the process of classroom learning through a classroom camera; Step two, preprocessing the classroom video data obtained in step one; Step three, binarizing the image after preprocessing in step two, selecting an interest region in each frame of the video stream, and extracting student behavior features; Step four, extracting student expression feature vectors and body feature vectors in the classroom using the local binary method LBP and the optical flow histogram method HOFO respectively on the processed classroom video stream data; Step five, inputting the obtained student expression feature vectors and body feature vectors into a one-class support vector machine OCSVM classifier for classification to obtain the classification result of whether the student behavior in the classroom is abnormal; The one-class support vector machine OCSVM aims to establish a decision boundary with the maximum distance between the normal data set and the origin; the best learning boundary is trained by the normal sample data set, including almost all normal behavior sample points; during testing, the points inside the boundary close to the origin belong to normal behavior, and the points outside the boundary belong to abnormal points; the OCSVM maps the input data to a high-dimensional feature space through a kernel function, so that it has better clustering, and then iteratively finds the optimal edge hyperplane to maximize the distance between the training data and the origin; Assume training data set Representing normal data set, to obtain the boundary, the model is optimized as follows: where w and p are parameters used to determine the decision boundary, which can be located in the optimization process; x i represents the training sample set, N is the total number of training data, v is the trade-off parameter, and ξ i n is the slack variable of x i , and the distance between the sample set and the coordinate origin is calculated. represents the nonlinear mapping from the data input space χ to the feature space , which is realized by kernel function transformation, and the learning of the OCSVM model is carried out in the feature space; the form of the kernel function is as follows:​ K(x,z)=(Φ(x)·Φ(z)), Where K(x,z) and Φ(x) represent the kernel function and the mapping function respectively; Φ(x)·Φ(z) is the inner product of Φ(x) and Φ(z); x,z∈χ, representing the sample of the data input space; The solution of the optimization model is obtained by Lagrange Multiplier, where α i is the Lagrange operator, let α = [α1, α2,..., α N ] T ; w is defined by ; The boundary parameter can be calculated by any x j , and then the best boundary is determined by support vector expansion. The decision function is as follows: wherein a i represents a Lagrangian operator, x i represents a non-zero training sample, and f(x) represents a decision function; Step six, logically fusing the two modal classification results of expression and body to obtain the final classroom abnormal behavior detection result.

2. The method of claim 1, wherein, In step two, the preprocessing includes image graying, video image resolution adjustment, video image size normalization, and video noise removal; The image graying refers to replacing the R, G, and B component values of a color image with a numerical value to represent the color depth of the image pixel point, with the range of values being [0, 255]; The video image resolution is adjusted to 256*1169 to ensure clear images; The video image size normalization refers to adjusting different video images to the same size in terms of pixel value; The video noise removal refers to removing unnecessary or redundant information in the image data.

3. The method of claim 1, wherein, In step three, the binarization refers to converting the image to a representation mode of only black and white, with the color depth of the pixel point being 0 or 255; the interest region in the video is selected using the maximum inter-class difference method.

4. The method of claim 1, wherein, In step four, the local binary method LBP calculates the pixel value by comparing the gray value of the center pixel P of the image with the gray values of the adjacent pixels, and the formula is as follows: L ~ l(s(n0-n c ),...,s(n7-n c )), where L represents a value obtained by calculation, l(x) represents a single s(x) calculation result, n0represents a pixel value other than the center pixel in the window, and n represents the number of pixels other than the center pixel in the window. c represents a center pixel value, 5. The method of claim 1, wherein, In step four, the descriptor calculation in HOFO is in block unit, including b w ×b h cells, each cell contains c w ×c h pixels, each pixel in the cell is mapped to a fixed angle range by the optical flow direction, and the statistical histogram is used for statistics to obtain the optical flow histogram of the whole cell. Then several cells are combined into a block, and the feature descriptors of all cells in the block are concatenated to obtain the HOFO feature descriptor of the block.

6. The method of claim 1, wherein, In step five, the single classification support vector machine is a special variant of the general support vector machine, which only needs to use one class of samples for training; the expression anomaly detector and the body movement anomaly detector corresponding to the expression feature vector and the body movement feature vector are set and trained respectively, namely the expression anomaly classification support vector machine and the body movement anomaly classification support vector machine; the expression anomaly includes crying, disgust, glassy eyes, and ghost expression; the body movement anomaly includes eating, playing mobile phone, pushing with hands, punching, standing, waving hands, and yawning.

7. The method of claim 1, wherein, In step six, the facial abnormal behavior detection result and the body abnormal behavior detection result are fused to output a final classroom abnormal behavior detection result; R e and R l respectively represent the facial abnormal behavior detection result and the body abnormal behavior detection result, and R represents the final detection result, R = R e ∩R l , wherein R represents the overall detection result; when R = 1, the student behavior is normal, i.e. R e = 1, R l = 1; when any one of the expression detection and the body movement detection is abnormal, the overall result is abnormal, i.e. R = 0.

8. A detection system implementing the method according to any one of claims 1 to 7, characterized in that, The system comprises a classroom learning behavior database module, an expression anomaly detection module, a body movement anomaly detection module, and a decision fusion module. The classroom learning behavior database module is used for collecting student classroom video data, determining a student classroom learning expression data set and a body movement data set, pre-processing the training data set, selecting a region of interest in a video image by borrowing a method in an OpenCV library, saving the processed data set in a classroom learning video database, and marking the data set with a normal or abnormal label. The expression anomaly detection module inputs a to-be-detected expression into a trained model to obtain a result of whether the expression is abnormal. The body movement anomaly detection module inputs a to-be-detected body movement into a trained model to obtain a result of whether the body movement is abnormal. The decision fusion module adopts a decision-level fusion strategy to logically fuse the expression anomaly detection result and the body movement anomaly detection result, and judges a final behavior anomaly detection result.

Images