Method and system for identifying abnormal student behavior in classroom based on video target detection

Abstract

The present invention relates to the field of video image processing methods, and discloses a method and system for identifying abnormal student behavior in a classroom based on video target detection, including student identity recognition, student abnormal behavior recognition, and obtaining identity information of students who have exhibited abnormal behavior. The present invention uses a single model to simultaneously obtain the behavior category and behavior trajectory of each student in the classroom, thereby reducing the amount of calculation. The present invention utilizes a feature enhancement strategy to improve the target behavior location and behavior recognition accuracy, and can perceive the student behavior trajectory, thereby improving the recognition accuracy of the identity of students with abnormal behavior.

Description

Technical Field

[0001] The present invention relates to the field of video image processing methods, and in particular to a method and system for identifying abnormal student classroom behavior based on video target detection. Background Art

[0002] In class, students' behavior status will reflect their learning status. In order to better understand their learning status, it is necessary to identify their abnormal behavior in class.

[0003] There are two existing methods for identifying abnormal student behavior in the classroom:

[0004] 1. Based on single frame image:

[0005] The single-frame image-based solution first obtains the location information of the target of interest through the target detection algorithm, and then uses the behavior recognition algorithm to obtain the behavior results of the target of interest.

[0006] 2. Based on video sequence

[0007] First, we need to obtain the position of the target of interest in each frame of the entire video sequence, construct an image set of the target of interest based on the position, and then feed the image set of the target of interest into the video-based behavior classification network.

[0008] The technical defects of existing identification methods are as follows:

[0009] Behavior recognition based on single-frame images and video sequences requires first using a target detection algorithm to locate the target of interest, and then using a classification algorithm for each target of interest to obtain the target behavior, which is computationally expensive. In addition, in methods based on single-frame images, due to motion afterimages, occlusions, or camera defocus, it is difficult to determine the target behavior location and behavior category information based on a single-frame image. Summary of the Invention

[0010] In order to solve the above technical problems, the present invention provides a method and system for identifying abnormal student behavior in classroom based on video target detection.

[0011] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0012] A method for identifying abnormal student behavior in a classroom based on video target detection includes the following steps:

[0013] Step 1: Obtain the student class video in real time, use the face detection algorithm to obtain the face position of each student in the student class video; use the face feature extraction algorithm to obtain the facial features of each student's face, and compare it with the student face feature library to obtain the identity corresponding to each face;

[0014] Step 2: Obtain the behavior position, initial behavior category, and behavior score of each student in each frame of the video;

[0015] Step 3: For each student ST in each frame image j , will students ST j The current behavior position is enlarged S times as the candidate region, and the intersection and union matching is performed with other frame images in the candidate region to obtain the student ST j Based on the matching behavior positions, initial behavior categories, and behavior scores of other frame images, the frequency of each abnormal behavior in all frame images is counted, and the abnormal behavior A with the highest frequency is found. It is then determined whether the number of times the student's abnormal behavior A occurs in all frame images exceeds M times; if so, it is considered that the student ST j Abnormal behavior A occurs, and the behavior location of abnormal behavior A is obtained; if not, it is considered that the student ST j No abnormal behavior A occurs; 1≤j≤p, where p is the total number of students in the class;

[0016] Step 4: For students who have abnormal behavior A, ST j In each frame, find the behavior position B corresponding to the abnormal behavior A with the highest behavior score; find the faces that intersect with position B in step 1, and calculate the distance between each face that intersects and position B. The face with the smallest distance corresponds to the identity of the student ST. j identity.

[0017] Step 2 specifically includes:

[0018] Get 2N frames of images every T seconds from the student class video;

[0019] The data of 2N frames of images (2N*3*H0*W0) is fed into the resnet50 backbone network to obtain the features of 2N*3*H1*W1. The features corresponding to the i-th frame of image are recorded as F i ,i=1,2,....2N, the size is C1*H1*W1; where H0 and W0 are the height and width of each frame image respectively, H1, W1, C1 are the feature F i The height, width and number of channels; i=1,2,....2N;

[0020] For each frame image, the cross attention method is used to aggregate features with the other 2N-1 frames respectively, and 2N-1 enhanced features are obtained; the 2N-1 enhanced features are summed up at the pixel level to obtain the enhanced features of the frame image.

[0021] For each target frame image, the enhanced features Use the classification detection head and regression detection head decoupled by yolox to obtain the behavior position of each student in each target frame image Initial category of behavior Behavioral scores

[0022] A system for identifying abnormal student behavior in classrooms based on video target detection, comprising:

[0023] Face recognition module: Real-time video of students in class is obtained, and the face detection algorithm is used to obtain the face position of each student in the video; the facial feature extraction algorithm is used to obtain the facial features of each student's face, and then compared with the student face feature library to obtain the identity corresponding to each face;

[0024] Behavior recognition module: obtains the behavior position, initial behavior category, and behavior score of each student in each frame of the class video;

[0025] Abnormal behavior recognition module: For each student ST in each frame image j , will students ST j The current behavior position is enlarged S times as the candidate region, and the intersection and union matching is performed with other frame images in the candidate region to obtain the student ST j Based on the matching behavior positions, initial behavior categories, and behavior scores of other frame images, the frequency of each abnormal behavior in all frame images is counted, and the abnormal behavior A with the highest frequency is found. It is then determined whether the number of times the student's abnormal behavior A occurs in all frame images exceeds M times; if so, it is considered that the student ST j Abnormal behavior A occurs, and the behavior location of abnormal behavior A is obtained; if not, it is considered that the student ST j No abnormal behavior A occurs; 1≤j≤p, where p is the total number of students in the class;

[0026] Identity determination module: For students who have abnormal behavior j In each frame, find the behavior position B corresponding to the abnormal behavior A with the highest behavior score; find the faces that intersect with position B in step 1, and calculate the distance between each face that intersects and position B. The face with the smallest distance corresponds to the identity of the student ST. j identity.

[0027] Compared with the prior art, the beneficial technical effects of the present invention are:

[0028] 1. The classroom behavior algorithm model based on video target detection proposed in this invention can simultaneously obtain the behavior category and behavior trajectory of each student in the classroom using a single algorithm model, which can reduce the amount of calculation.

[0029] 2. For multiple frames of images, the backbone network of the video target detection model proposed in the present invention is shared, which can reduce the amount of calculation.

[0030] 3. The present invention utilizes feature enhancement strategy to improve the accuracy of target behavior location and behavior recognition.

[0031] 4. The method proposed in the present invention for obtaining target student behavior and behavior trajectory information can improve the accuracy of student behavior recognition and perceive the trajectory of student behavior.

[0032] 5. The method for obtaining the identity information of students who have committed abnormal behavior proposed in the present invention can improve the accuracy of identifying the identities of students with abnormal behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Flowchart of the abnormal behavior identification method of the present invention. DETAILED DESCRIPTION

[0034] A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[0035] like Figure 1 As shown, the method for identifying abnormal student behavior in a classroom based on video target detection in the present invention includes the following steps.

[0036] S1. Student identity identification:

[0037] After the class begins, real-time video data is captured and a face detection algorithm is used to determine the facial location of each student. For each student's face, a facial feature extraction algorithm is used to extract facial features. This information is then compared with a database of student facial features to obtain the corresponding identity information. This step allows for the acquisition of both the location and identity of each face.

[0038] S2. Identification of abnormal student behavior:

[0039] In practice, it's difficult to determine the target behavior's location and category based on a single frame due to motion artifacts, occlusions, or camera defocus. For a continuous video, we can enhance the features of the target frames by incorporating contextual semantic information, accurately identifying the location and category of each target behavior in each frame.

[0040] The abnormal behavior recognition model in this invention can simultaneously monitor and identify each student's abnormal behavior in the classroom, such as playing with their phone or sleeping. The abnormal behavior recognition model in this invention includes a backbone network module, a feature enhancement module, a classification detection head, and a regression detection head. The specific steps are as follows:

[0041] 1. Obtain the behavior location, initial behavior category information, and behavior score of each student in each frame of the image through the abnormal behavior recognition model.

[0042] 1) Obtain 2N frames of image every T seconds from real-time video data.

[0043] 2) Send the 2N frame image data 2N*3*H0*W0 to the resnet50 backbone network to obtain 2N*3*H1*W1 features. The feature corresponding to the i-th target frame image is recorded as F i ,i=1,2,....2N, the size is C1*H1*W1, where H0 and W0 are the height and width of each frame image respectively, H1, W1, C1 are the feature F i Height, width and number of channels;.

[0044] 3) For each frame image, the cross-attention method is used to perform feature aggregation with the other 2N-1 frames respectively, and 2N-1 enhanced features are obtained; the 2N-1 enhanced features are summed at the pixel level to obtain the enhanced features of the frame image

[0045] 4) Enhanced features for each target frame image Use the classification detection head and regression detection head decoupled by yolox to obtain the behavior position of each student in each target frame image Initial category of behavior Behavioral scores The set composed of j=1,2,......p, where p is the total number of students in the class.

[0046] 2. Obtain information on target students’ behavior and behavior trajectory

[0047] In classroom scenarios, students have a limited range of activities. Therefore, for each target student in each target frame image, its current behavior position is used to expand S times as a candidate area, and intersection-over-union (IOU) matching is performed with other frame images within the candidate area to obtain the matching position information of the current target student in the other 2N-1 frame images and the corresponding initial behavior category information and behavior score information. The frequency of occurrence of each behavior in the 2N frames is counted, and the behavior A with the highest frequency is found. It is determined whether the number of times the target student's behavior A occurs in the 2N frames exceeds M times. If so, it can be defined that the current target student has performed behavior A within the past T time. At the same time, the trajectory information of the current target student's behavior A within the past T time can be obtained, thereby improving the accuracy of student behavior recognition and perceiving the trajectory of student behavior. If not, it is considered that the target student has no abnormal behavior in the classroom within the past T time.

[0048] S3. Obtain the identity information of students who have committed abnormal behavior:

[0049] For the target student who exhibited abnormal behavior, find the behavior location B corresponding to the abnormal behavior with the highest behavior score in each frame. Find the face that intersects with B from step 1 and calculate the distance between the intersecting face and location B. The face with the smallest distance is the target student's identity. The distance here generally refers to the center distance.

[0050] The present invention proposes an end-to-end method for identifying abnormal behavior in the classroom based on video target detection, which can obtain the target student's behavior and behavior trajectory information, obtain the identity of the student who has committed abnormal behavior, and use feature enhancement strategies to improve the target behavior location and behavior recognition accuracy.

[0051] The system of the present invention corresponds to the method, and the preferred embodiments of the method are also applicable to the system.

[0052] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. It is intended that all variations within the meaning and range of equivalents of the claims be embraced herein, and any reference signs in the claims should not be construed as limiting the claims to which they relate.

[0053] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A method for identifying abnormal student behavior in a classroom based on video target detection, comprising the following steps: Step 1: Obtain the student class video in real time, use the face detection algorithm to obtain the face position of each student in the student class video; use the face feature extraction algorithm to obtain the facial features of each student's face, and compare it with the student face feature library to obtain the identity corresponding to each face; Step 2: Obtain the behavior position, initial behavior category, and behavior score of each student in each frame of the video; Step 3: For each student ST in each frame image j , will students ST j The current behavior position is enlarged S times as the candidate region, and the intersection and union matching is performed with other frame images in the candidate region to obtain the student ST j Based on the matching behavior positions, initial behavior categories, and behavior scores of other frame images, the frequency of each abnormal behavior in all frame images is counted, and the abnormal behavior A with the highest frequency is found. It is then determined whether the number of times the student's abnormal behavior A occurs in all frame images exceeds M times; if so, it is considered that the student ST j Abnormal behavior A occurs, and the behavior location of abnormal behavior A is obtained; if not, it is considered that the student ST j No abnormal behavior A occurs; 1≤j≤p, where p is the total number of students in the class; Step 4: For students who have abnormal behavior A, ST j In each frame, find the behavior position B corresponding to the abnormal behavior A with the highest behavior score; find the faces that intersect with position B in step 1, and calculate the distance between each face that intersects and position B. The face with the smallest distance corresponds to the identity of the student ST. j identity.

2. The method for identifying abnormal student behavior in a classroom based on video target detection according to claim 1 is characterized in that: Step 2 specifically includes: Get 2N frames of images every T seconds from the student class video; The data of 2N frames of images (2N*3*H0*W0) is fed into the resnet50 backbone network to obtain 2N*3*H1*W1 features. The feature corresponding to the i-th frame image is recorded as F i , i=1,2,....2N,the size is C1*H1*W1;where H0 and W0 are the height and width of each frame image respectively, H1, W1, C1 are the feature F i The height, width and number of channels; i = 1, 2, .... 2N; For each frame image, the cross attention method is used to aggregate features with the other 2N-1 frames respectively, and 2N-1 enhanced features are obtained; the 2N-1 enhanced features are summed up at the pixel level to obtain the enhanced features of the frame image. For each target frame image, the enhanced features Use the classification detection head and regression detection head decoupled by yolox to obtain the behavior position of each student in each target frame image Initial category of behavior Behavioral scores 3. A system for identifying abnormal student behavior in classroom based on video target detection, characterized by: include: Face recognition module: Real-time video of students in class is obtained, and the face detection algorithm is used to obtain the face position of each student in the video; the facial feature extraction algorithm is used to obtain the facial features of each student's face, and then compared with the student face feature library to obtain the identity corresponding to each face; Behavior recognition module: obtains the behavior position, initial behavior category, and behavior score of each student in each frame of the class video; Abnormal behavior recognition module: For each student ST in each frame image j , will students ST j The current behavior position is enlarged S times as the candidate region, and the intersection and union matching is performed with other frame images in the candidate region to obtain the student ST j Based on the matching behavior positions, initial behavior categories, and behavior scores of other frame images, the frequency of each abnormal behavior in all frame images is counted, and the abnormal behavior A with the highest frequency is found. It is then determined whether the number of times the student's abnormal behavior A occurs in all frame images exceeds M times; if so, it is considered that the student ST j Abnormal behavior A occurs, and the behavior location of abnormal behavior A is obtained; if not, it is considered that the student ST j No abnormal behavior A occurs; 1≤j≤p, where p is the total number of students in the class; Identity determination module: For students who have abnormal behavior j In each frame, find the behavior position B corresponding to the abnormal behavior A with the highest behavior score; find the faces that intersect with position B in step 1, and calculate the distance between each face that intersects and position B. The face with the smallest distance corresponds to the identity of the student ST. j identity.

Images