Attendance detection method and system based on offline classroom monitoring

Abstract

The present invention discloses an attendance rate detection method and system based on offline classroom monitoring, wherein the method includes training a head-body joint detection network model, optimizing the weight of the model by using weighted joint loss; obtaining offline classroom monitoring video, extracting key frame images at intervals of a certain number of frames; inputting the key frame images into the trained head-body joint detection network model to obtain a head candidate frame; using a post-processing algorithm to remove duplicate frames, background frames and unreasonable frames in the candidate frame, thereby obtaining the number of people contained in the current key frame; dividing the number of people detected in the key frame by the number of people who should attend the class to obtain the attendance rate corresponding to this key frame. This method can obtain the number of people present, and then obtain the attendance rate corresponding to this key frame. This detection method has high accuracy, real-time performance and automation, and can help managers quickly understand the real-time distribution of classroom attendance.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an attendance rate detection method and system based on offline classroom monitoring. Background Art

[0002] There are many types of indicators for offline classroom quality assessment, such as attendance rate, head-up rate, front-row seating rate, abnormal behavior detection, etc. Among them, attendance rate is a very important indicator. The level of attendance rate represents the students' enthusiasm and interest in the class to a certain extent, and it can also be used as an important reference index for evaluating the level of teachers. At present, there are many commonly used attendance rate statistics methods, but they all have certain drawbacks. For example, offline roll call will take up class time, face attendance machines, fingerprint punching machines, etc. require additional budget to purchase equipment, and WeChat applet punching may have the situation of signing in on behalf of others and taking photos to sign in remotely. These statistics are generally only carried out before class. When students leave early during class, they cannot be detected, and the attendance rate cannot be obtained dynamically and in real time. In the prior art, attendance rate detection methods based on live body recognition and image analysis are proposed. Although these methods can automatically count classroom videos, they need to set up front modules such as manual feature extraction and multiple area detection in series. The accuracy of the front module has a great impact on subsequent detection, making the method complicated and unable to guarantee detection efficiency. Summary of the invention

[0003] In view of the problems existing in the prior art, the present invention discloses an attendance rate detection method based on offline classroom monitoring, which specifically includes the following steps:

[0004] Train the head and human body joint detection network model and use weighted joint loss to optimize the model weights;

[0005] Obtain offline classroom surveillance videos and extract key frame images at a certain interval;

[0006] Input the key frame image into the trained head-human joint detection network model to obtain the head candidate frame;

[0007] A post-processing algorithm is used to remove duplicate frames, background frames and unreasonable frames in the candidate frames, so as to obtain the number of people contained in the current key frame;

[0008] Divide the number of people detected in the key frame by the number of people who should be in class to obtain the attendance rate corresponding to this key frame.

[0009] Furthermore, when training the head and human body joint detection network model:

[0010] According to the principle that the head and the upper body of the human body appear at the same time and in a certain proportion, the corresponding human body frame is automatically generated in proportion according to each head frame to obtain a data set with head frame and human body frame annotations, and the data set is divided into a training set and a validation set according to the proportion;

[0011] A human head and human body joint detection network model is established, which includes a backbone network and an FPN module. The backbone network is used to extract feature response maps at different resolutions, including a series of continuous CPM modules and CBL modules. The FPN module performs upsampling and summation processing on the feature response maps.

[0012] The training set is input into the head-body joint detection network model, feature information of different resolutions is extracted and fused to output the candidate frame coordinates of the head frame and the body frame;

[0013] The actual coordinates of the head frame and the human body frame are compared with the candidate frames predicted by the model to obtain the weighted joint loss, and the weights of the model are optimized based on the weighted joint loss.

[0014] Furthermore, when the weighted joint loss is used to optimize the weights of the head and human joint detection network model, the weighted joint loss is expressed as:

[0015]

[0016] Among them, i represents the i-th box, the value of m is 0 and 1, representing the head and the human body respectively, p is the category probability of the candidate box, and t is the coordinate information of the candidate box;

[0017] The formula for calculating the loss of heads and bodies is as follows:

[0018]

[0019] Among them, L c is the logarithmic loss of classification, L r is the mean square error of the coordinates, p*, t* represent the category probability and coordinate information of the true label respectively, and λ m , α m Represent the corresponding weights respectively.

[0020] The model weights are optimized using stochastic gradient descent according to the weighted joint loss.

[0021] An attendance rate detection method system based on offline classroom monitoring, comprising:

[0022] The training module trains the head and human body joint detection network model and optimizes the model weights using weighted joint loss;

[0023] Key frame extraction module, used to obtain key frame images of classroom videos;

[0024] Head detection module, used to detect head candidate frames in key frames;

[0025] The post-processing module is used to remove duplicate frames, background frames and unreasonable frames in the candidate frames to obtain the number of heads in the key frame;

[0026] The output module calculates the attendance rate based on the number of people detected in the key frame and outputs it.

[0027] Due to the adoption of the above technical scheme, the present invention provides a method and system for detecting attendance rate based on offline classroom monitoring, wherein the method first trains a head-body joint detection network model, and optimizes the weight of the model by using a weighted joint loss; obtains offline classroom monitoring video, and extracts key frame images at intervals of a certain number of frames; inputs the key frame images into the trained head-body joint detection network model to obtain a head candidate frame; adopts a post-processing algorithm to remove duplicate frames, background frames and unreasonable frames in the candidate frame to obtain the number of people contained in the current key frame; divides the number of people detected in the key frame by the number of people who should attend the class to obtain the attendance rate corresponding to this key frame. The neural network used in the present invention is based on a semi-supervised head and body joint detection model. The head does not exist in isolation, and is usually accompanied by human body information such as shoulders. When the texture features of the head cannot be fully extracted, these human body information can be used to assist in detecting the location of the head. The present invention only needs a data set with head annotations, and can automatically generate human body annotation frames by a semi-supervised method, and use the human body frame to assist in detecting the head frame, which can solve the problems of small targets, blur, and severe occlusion of the head that are difficult to detect. The present invention can detect the attendance of the whole class in real time and reflect situations such as lateness and early departure. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 is a flow chart of the method of the present invention;

[0030] Figure 2 is a flow chart of the system of the present invention;

[0031] Figure 3 This is a structural diagram of the human body joint detection network model in the present invention;

[0032] Figure 4 This is a schematic diagram of the CBL module in the present invention;

[0033] Figure 5 This is a schematic diagram of the CPM module in the present invention;

[0034] Figure 6 It is a schematic diagram of the FPN module in the present invention. DETAILED DESCRIPTION

[0035] In order to make the technical solutions and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention:

[0036] like Figure 1 The attendance rate detection method based on offline classroom monitoring shown includes the following steps:

[0037] S1: Train the head-body joint detection network model and use weighted joint loss to optimize the model weights. The head-body joint detection network model training process specifically includes the following steps:

[0038] It is necessary to obtain the data set first and then train the head and human joint detection network model. The model structure is as follows Figure 3 shown.

[0039] S1.1: Get the head dataset

[0040] Obtain a dataset from a public database, or manually annotate a dataset with head frame coordinates. Under normal circumstances, the head and upper body will appear at the same time and in a regular pattern. Therefore, a series of proportional frames can be used to simulate the real head frame and body frame to obtain a dataset with head frame and body frame annotations, and divide the dataset into a training set and a validation set in proportion.

[0041] Assuming that the coordinates of the face frame obtained from the data set are t0, the coordinates of the human body frame can be defined as t1, which can be expressed as:

[0042] t0=(x0,y0,w0,h0)

[0043]

[0044] Wherein, x0, y0, w0, h0 represent the distance from the center point of the face frame to the x-axis and y-axis, as well as the width and height information, respectively. It is worth noting that the t1 coordinate ratio set in the present invention is for reference only and can be modified according to actual needs.

[0045] S1.2: Extracting feature response maps using backbone networks

[0046] The training set is input into the backbone network. After several consecutive processings by CPM and CBL modules, feature response maps at different resolutions can be obtained. If the size of the input map is 640x640, features of sizes of 320x320, 160x160, 80x80, 40x40, and 20x20 can be obtained respectively, which can be used as the input of subsequent modules.

[0047] Among them, the CBL module consists of a 3x3 convolution layer Conv, a batch normalization BN layer, and a LeakyRelu activation layer, such as Figure 4 The design of CPM (Context Predict Module) draws on the idea of ​​Inception structure and consists of three branches. Each branch uses 1, 2, and 3 CBL modules for processing, so as to extract receptive field features of different sizes. Finally, the Concat operation is used to connect the features in parallel to make the network wider and obtain more receptive field information.

[0048] S1.3: Fusion of information at different scales

[0049] The present invention uses the FPN module to fuse information between different scales, but not all of the five different sizes of outputs of the backbone network are used as inputs of the FPN module, because the receptive field of the lower layer of the network is relatively small and it is impossible to extract complete head features. Therefore, only the FPN module used for features of 80x80 and 40x40 sizes is processed, and its function is to allow the network to learn both the features transmitted from low order to high order and the features transmitted from high order to low order. Taking the feature map of 80x80 resolution as an example, the input 1 of FPN is the 80x80 feature of the current branch, and the input 2 is the 40x40 feature of the adjacent next branch. The input 2 is upsampled to obtain the 80x80 feature, and then added to the input 1 to obtain the final output. The FPN module is not used at the 20x20 resolution because it is already the deepest layer of the network and cannot obtain another branch with a smaller resolution, so it is replaced by the CBL module.

[0050] S1.4: Get candidate boxes and calculate weighted joint loss

[0051] After a series of feature extractions, a large number of candidate frames for head frames and human frames can be obtained. During the network training phase, the actual coordinates of the head frames and human frames need to be compared with the candidate frames predicted by the model to obtain a weighted joint loss, and the weights of the head and human joint detection network model are optimized using the stochastic gradient descent method. The weighted joint loss is expressed as:

[0052]

[0053] Among them, i represents the i-th frame, m takes the value of 0 and 1, representing the head and human body respectively, p is the category probability of the candidate frame, and t is the coordinate information of the candidate frame. The loss calculation formula for the head and human body is as follows:

[0054]

[0055] Among them, L c is the logarithmic loss of classification, L r is the mean square error of the coordinates, p*, t* represent the category probability and coordinate information of the true label respectively, and λ m , α m They represent the corresponding weights respectively and can be set and adjusted manually.

[0056] S2: Set a specific interval frame value, extract images from the offline classroom monitoring video at a specific frame rate and save them as key frames for subsequent algorithm detection. The reason for extracting key frames is that a normal class is generally 45 minutes, and each minute of video corresponds to thousands of continuous frames. The similarity between these images is very high. It is not necessary to detect attendance for each frame, which is time-consuming and labor-intensive.

[0057] S3: Input the key frame image into the trained head-human joint detection network model to obtain the head candidate frame. The trained head-human joint detection network model can output the head frame and the candidate frame of the human frame during detection. The role of the human frame is to provide auxiliary information in the training stage. In the detection stage, only the head frame can be used for the subsequent attendance rate calculation.

[0058] S4: Post-processing to obtain the head detection result. Use the NMS algorithm to post-process it, remove redundant background boxes and repeated prediction boxes, and get the final result. NMS is the full name of non-maximum suppression algorithm, which can remove non-maximum values ​​in local areas. In head detection, each candidate box has a corresponding confidence level. Assuming that the candidate box set is T and the screening result is R, the detection process of NMS is to select the box A with the highest confidence level in T, delete it from T, and add it to R. Calculate the IOU (intersection-over-union ratio) of A and other candidate boxes in T, and delete the boxes in T whose IOU is greater than a certain threshold. For the remaining boxes in T, continue to repeat the above process until T is empty, and then R saves the final result.

[0059] S5: Calculate the attendance rate. Divide the number of people detected in each key frame by the number of people who should be present in class to get the attendance rate corresponding to this key frame. The results of all key frames are the attendance of each period of the whole class, which can clearly show whether there are phenomena such as lateness and early departure.

[0060] The embodiment of the present invention provides an attendance rate detection system based on offline classroom monitoring, including a training module for training a head-body joint detection network model and optimizing the weight of the model using a weighted joint loss; a key frame extraction module for obtaining key frame images of classroom videos; a head detection module for detecting head candidate frames in key frames; a post-processing module for removing duplicate frames, background frames, and unreasonable frames in the candidate frames to obtain the number of heads in the key frames; and an output module for calculating the attendance rate and outputting it. For specific implementation, refer to the method embodiment.

[0061] The present invention proposes an attendance rate detection method and system based on offline classroom monitoring, trains a head-human joint detection network model, and uses weighted joint loss to optimize the weight of the model. For a video of a certain classroom, key frame images are extracted at intervals of a specific number of frames, and the trained head-human joint detection network model is used to detect the image to obtain a large number of head candidate frames. Then, a post-processing method is used to filter the repeated frames and background frames to obtain the number of attendees, and then the attendance rate corresponding to the key frame is obtained. This detection method has high accuracy, real-time performance, and automation, and can help managers quickly understand the real-time distribution of classroom attendance.

[0062] The human body frame used in training the joint detection model of the present invention is automatically generated, and a higher-precision model can be trained on the basis of greatly saving manual labeling costs.

[0063] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for detecting attendance rate based on offline classroom monitoring, characterized in that include: Train the head and human body joint detection network model and use weighted joint loss to optimize the model weights; Obtain offline classroom surveillance videos and extract key frame images at a certain interval; Input the key frame image into the trained head-human joint detection network model to obtain the head candidate frame; A post-processing algorithm is used to remove duplicate frames, background frames and unreasonable frames in the candidate frames, so as to obtain the number of people contained in the current key frame; Divide the number of people detected in the key frame by the number of people who should be in class to obtain the attendance rate corresponding to this key frame; When training the head-body joint detection network model: According to the principle that the head and the upper body of the human body appear at the same time and in a certain proportion, the corresponding human body frame is automatically generated in proportion according to each head frame to obtain a data set with head frame and human body frame annotations, and the data set is divided into a training set and a validation set according to the proportion; A human head and human body joint detection network model is established, the model includes a backbone network and an FPN module, wherein the backbone network is used to extract feature response maps at different resolutions, including a series of continuous CPM modules and CBL modules, and the FPN module performs upsampling and summing processing on the feature response maps; The CBL module consists of a 3x3 convolutional layer Conv, a batch normalization BN layer, and a Leaky Relu activation layer; the CPM module consists of three branches, each of which is processed by 1, 2, and 3 CBL modules respectively to extract receptive field features of different sizes. Finally, the Concat operation is used to connect the features in parallel to make the network wider and obtain more receptive field information. The training set is input into the head-body joint detection network model, feature information of different resolutions is extracted and fused to output the candidate frame coordinates of the head frame and the body frame; Compare the actual coordinates of the head frame and the human body frame with the candidate frames predicted by the model to obtain the weighted joint loss, and optimize the model weights based on the weighted joint loss; When the weighted joint loss is used to optimize the weights of the head and human joint detection network model, the weighted joint loss is expressed as: Among them, i represents the i-th box, the value of m is 0 and 1, representing the head and the human body respectively, p is the category probability of the candidate box, and t is the coordinate information of the candidate box; The formula for calculating the loss of heads and bodies is as follows: Among them, L c is the logarithmic loss of classification, L r is the mean square error of the coordinates, p*, t* represent the category probability and coordinate information of the true label respectively, and λ m , α m Respectively represent the corresponding weights; The model weights are optimized using stochastic gradient descent according to the weighted joint loss.

2. An attendance rate detection system based on offline classroom monitoring, characterized in that: include: The training module trains the head and human body joint detection network model and optimizes the model weights using weighted joint loss; Key frame extraction module, used to obtain key frame images of classroom videos; Head detection module, used to detect head candidate frames in key frames; The post-processing module is used to remove duplicate frames, background frames and unreasonable frames in the candidate frames to obtain the number of heads in the key frame; The output module calculates the attendance rate based on the number of people detected in the key frame and outputs it; When training the head-body joint detection network model: According to the principle that the head and the upper body of the human body appear at the same time and in a certain proportion, the corresponding human body frame is automatically generated in proportion according to each head frame to obtain a data set with head frame and human body frame annotations, and the data set is divided into a training set and a validation set according to the proportion; A human head and human body joint detection network model is established, the model includes a backbone network and an FPN module, wherein the backbone network is used to extract feature response maps at different resolutions, including a series of continuous CPM modules and CBL modules, and the FPN module performs upsampling and summing processing on the feature response maps; The CBL module consists of a 3x3 convolutional layer Conv, a batch normalization BN layer, and a Leaky Relu activation layer; the CPM module consists of three branches, each of which is processed by 1, 2, and 3 CBL modules respectively to extract receptive field features of different sizes. Finally, the Concat operation is used to connect the features in parallel to make the network wider and obtain more receptive field information. The training set is input into the head-body joint detection network model, feature information of different resolutions is extracted and fused to output the candidate frame coordinates of the head frame and the body frame; Compare the actual coordinates of the head frame and the human body frame with the candidate frames predicted by the model to obtain the weighted joint loss, and optimize the model weights based on the weighted joint loss; When the weighted joint loss is used to optimize the weights of the head and human joint detection network model, the weighted joint loss is expressed as: Among them, i represents the i-th box, the value of m is 0 and 1, representing the head and the human body respectively, p is the category probability of the candidate box, and t is the coordinate information of the candidate box; The formula for calculating the loss of heads and bodies is as follows: Among them, L c is the logarithmic loss of classification, L r is the mean square error of the coordinates, p*, t* represent the category probability and coordinate information of the true label respectively, and λ m , α m Respectively represent the corresponding weights; The model weights are optimized using stochastic gradient descent according to the weighted joint loss.

Images