Classroom activity identification method and device and classroom monitoring system

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a classroom activity identification method and device and a classroom monitoring system. The method comprises the following steps: acquiring monitoring images of a plurality of classrooms with different scales; wherein each monitoring image is provided with a label of a corresponding activity type; extracting feature data of the monitoring image, and constructing a training data set; training an activity recognition model by using the training data set; detecting target feature data by using the trained activity recognition model to obtain a recognition result, and generating output information based on the recognition result; and obtaining feedback information of the output information, and performing incremental training on the activity recognition model based on a new activity type under the condition that the feedback information comprises the unrecognized new activity type. By adopting the method, the types of various dynamically changing activities in the classroom can be effectively identified, and the newly appearing activity type can be quickly adapted through a small number of labeled samples.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a classroom activity recognition method, device and classroom monitoring system. Background Art

[0002] With the rapid development of artificial intelligence technology, analyzing student activities in the classroom through artificial intelligence technology can help to adaptively adjust teaching content and methods and improve teaching effectiveness. Especially when there are a large number of students, traditional teaching methods are difficult to pay attention to the personalized needs of individual students, making it difficult for teachers to provide personalized teaching guidance. Therefore, it is necessary to accurately identify and analyze student activities in the classroom in order to improve teaching content or methods according to the types of student activities and improve teaching quality.

[0003] Existing activity recognition solutions based on classroom monitoring rely on a large amount of labeled data and static model training. The training process consumes a lot of time and computing resources, and the generalization ability of the model is low. It can only cope with preset fixed activity types and limited environments, and cannot adapt to new activity types. However, the dynamic changes of student activities in the classroom environment are relatively frequent, and the model needs to be retrained every time a new activity type appears. Based on this, how to quickly adapt to new activity types in the classroom environment, improve the generalization ability of the model and save training costs are the current problems that need to be solved. Summary of the invention

[0004] In view of this, the present application aims to propose a classroom activity recognition method, device and classroom monitoring system to improve the model's adaptability to new activity types emerging in the classroom environment and save model training costs.

[0005] To achieve the above purpose, the technical solution of this application is as follows:

[0006] A first aspect of an embodiment of the present application provides a method for identifying classroom activities, the method comprising:

[0007] Acquire surveillance images of multiple classrooms of different sizes; the surveillance images include: video frame images of different time periods, different viewing angles and different activity types for each classroom; wherein each surveillance image is labeled with a corresponding activity type;

[0008] Extracting feature data of the monitoring image and constructing a training data set;

[0009] Using the training data set to train an activity recognition model; the activity recognition model is constructed based on an adaptive covariance classifier;

[0010] Using the trained activity recognition model to detect target feature data, obtain a recognition result, and generate output information based on the recognition result; the target feature data is feature data generated based on a target monitoring image of a target classroom;

[0011] Acquire feedback information of the output information, and when the feedback information includes an unrecognized new activity type, perform incremental training on the activity recognition model based on the new activity type.

[0012] Optionally, the activity recognition model includes a first branch and a second branch; wherein the first branch is a neural network based on supervised contrastive learning, and the second branch is an adaptive covariance classifier; training the activity recognition model using the training data set includes:

[0013] Taking the feature data in the training data set as input data, inputting them into the first branch and the second branch respectively; processing the input data through the first branch to obtain a first feature; and processing the input data through the second branch to obtain a second feature;

[0014] Inputting the first feature and the second feature into a fully connected layer to obtain a recognition result;

[0015] Based on the cross entropy loss function and the contrast loss function, calculating the difference between the recognition result and the label of the input data;

[0016] Based on the difference, a gradient descent algorithm is used to adjust the parameters of the activity recognition model.

[0017] Optionally, incrementally training the activity recognition model based on the new activity type includes:

[0018] Adding a label of the new activity type to the target monitoring image, and adding the target feature data to the training data set;

[0019] The activity recognition model is incrementally trained using the training data set to adjust parameters of the first branch and the second branch.

[0020] Optionally, the feedback information further includes an activity type identification error; after obtaining the feedback information of the output information, the method further includes:

[0021] Determine the actual activity type of the target surveillance image and add a corresponding label;

[0022] Adding the target feature data to the training data set;

[0023] The activity recognition model is incrementally trained using the training data set to adjust parameters of the first branch.

[0024] Optionally, build a training dataset based on all surveillance images, including:

[0025] Performing data enhancement on each monitoring image to expand the number of monitoring images; the data enhancement includes: random rotation, scaling, cropping, changing brightness or contrast;

[0026] Convert all surveillance images to the same size and color space;

[0027] Extract feature data of all surveillance images as training data, and convert all training data to the same scale;

[0028] Build a training dataset based on all training data.

[0029] Optionally, feature data of all surveillance images are extracted as training data, including:

[0030] Using a pre-trained feature extraction model to extract feature data of all surveillance images as original features; the feature extraction model is constructed based on a neural network;

[0031] Extracting interaction features based on interaction information between people and between people and objects in the surveillance image;

[0032] The original features and the interactive features are used as training data.

[0033] Optionally, extracting feature data of all surveillance images also includes:

[0034] Filter the behaviors that are of interest to all current activity types and determine the behaviors whose weights are greater than the weight threshold;

[0035] Based on the screened behaviors, extracting corresponding key behavior features from the surveillance image;

[0036] The key behavioral features are added to the training data.

[0037] Optionally, extracting feature data of all surveillance images also includes:

[0038] Derived features are calculated based on the original features, and the derived features are added to the training data.

[0039] Optionally, generating output information based on the recognition result includes:

[0040] Obtaining person information and time corresponding to the recognition result;

[0041] Acquiring environmental information of the target classroom;

[0042] Output information is generated based on the recognition result, the person information, the time and the environment information.

[0043] According to a second aspect of an embodiment of the present application, a classroom activity recognition device is provided, which is used to implement the steps in the method provided in the first aspect of the embodiment of the present application, and the device includes:

[0044] A preprocessing module is configured to obtain surveillance images of multiple classrooms of different sizes; the surveillance images include: video frame images of different time periods, different viewing angles and different activity types for each classroom; wherein each surveillance image is labeled with a corresponding activity type; extract feature data of the surveillance images, and construct a training data set;

[0045] A training module, configured to train an activity recognition model using the training data set; the activity recognition model is constructed based on an adaptive covariance classifier;

[0046] The recognition module is configured to detect the target feature data using the trained activity recognition model to obtain a recognition result, and generate output information based on the recognition result; the target feature data is feature data generated based on the target monitoring image of the target classroom;

[0047] A feedback module is further configured to obtain feedback information of the output information;

[0048] The training module is further configured to, when the feedback information includes an unrecognized new activity type, perform incremental training on the activity recognition model based on the new activity type.

[0049] According to a third aspect of an embodiment of the present application, a classroom monitoring system is provided, including:

[0050] The acquisition unit includes: at least one camera deployed in each classroom, used to collect surveillance videos in each classroom; a recording module, used to record the time and environment information corresponding to each surveillance video, and store all surveillance videos in a video database;

[0051] The recognition unit comprises: a conversion module and a classroom activity recognition device as provided in the second aspect of the embodiment of the present application; the conversion module is used to convert the monitoring video into a monitoring image;

[0052] A video database, used to store the surveillance videos collected by the collection unit;

[0053] An activity database, used to store description information of each activity, the description information including: a detailed description of the activity, common behaviors and conditions for the activity to occur;

[0054] The pattern database is used to store the identification patterns corresponding to each activity type, and the identification patterns include: feature description, behavior parameters, identification methods and usage scenarios of the activity type.

[0055] Using the classroom activity recognition method provided by the present application, first obtain a small amount of labeled samples and construct a training data set. Specifically, obtain surveillance images collected in multiple classrooms of different sizes. For each classroom, collect video frame images of different time periods, different surveillance angles, and different activity types to construct a diverse training data set. Then, use the training data set to train the activity recognition model built based on the adaptive covariance classifier, so that the activity recognition model can accurately distinguish different activity types in different time periods and different surveillance angles, and meet the intelligent recognition of all current activity types. In the event that a new activity type is not recognized, the activity recognition model is incrementally trained based on user feedback information so that the model has the ability to recognize new activity types.

[0056] In this application, activity types are constructed based on an adaptive covariance classifier. Given that the adaptive covariance classifier has the characteristic of dynamically adjusting the decision boundary to adapt to the feature distribution of new categories, the constructed activity recognition model can dynamically adjust the model parameters with a small number of new samples through incremental training to adapt to the newly emerged activity types without retraining the model. Compared with the traditional method of training static models with a large amount of labeled data, this application can save a lot of time costs and computing resources, and can cope with the dynamic changes of activity types in the classroom environment. When a new activity type appears, it can be incrementally trained with a small number of new samples to quickly adapt to the new activity type. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0058] Figure 1 is a flow chart of a classroom activity recognition method proposed in one embodiment of the present application;

[0059] Figure 2 is a schematic diagram of an activity recognition model processing feature data in an embodiment of the present application;

[0060] Figure 3 is a flowchart of training an activity recognition model in one embodiment of the present application;

[0061] Figure 4 is a schematic diagram of a classroom activity recognition device proposed in an embodiment of the present application;

[0062] Figure 5 It is a schematic diagram of a classroom monitoring system proposed in one embodiment of the present application. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0064] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present application. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0065] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0066] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects as detailed in the present application.

[0067] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0068] The recognition of activity types in the classroom in the related art is usually unable to effectively cope with the dynamic changes of activity types and the emergence of new activities in classroom monitoring images. Moreover, in the related schemes, after the emergence of new activity types, retraining the model requires the collection of a large number of training samples, and it is difficult to accurately identify the new activity types when the number of new samples is limited. Moreover, retraining a new model may cause knowledge forgetting, resulting in a decrease in the recognition effect of old activity types. In view of this, this embodiment proposes an activity recognition model based on an adaptive covariance classifier, which can accurately recognize the real-time and dynamically changing activity types in the classroom by training the model, and quickly adapt to the emergence of new activities, and adapt to the dynamic changes of the classroom environment and student behavior.

[0069] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0070] Figure 1 : is a flow chart of a classroom activity recognition method proposed in one embodiment of the present application. Figure 1 As shown, the method includes:

[0071] S1: Acquire surveillance images of multiple classrooms of different sizes; the surveillance images include: video frame images of different time periods, different viewing angles and different activity types for each classroom; wherein each surveillance image is labeled with a corresponding activity type;

[0072] S2: extracting feature data of the monitoring image and constructing a training data set;

[0073] S3: training an activity recognition model using the training data set; the activity recognition model is constructed based on an adaptive covariance classifier;

[0074] S4: using the trained activity recognition model to detect the target feature data to obtain a recognition result, and generating output information based on the recognition result; the target feature data is feature data generated based on the target monitoring image of the target classroom;

[0075] S5: Obtain feedback information of the output information, and when the feedback information includes an unrecognized new activity type, perform incremental training on the activity recognition model based on the new activity type.

[0076] In this embodiment, a training data set is constructed based on classroom monitoring images of different sizes. First, monitoring images of classrooms of different sizes (such as small classrooms, large lecture halls, etc.) are obtained from the classroom monitoring system, including images of different time periods, different viewing angles and different activity types. Specifically, the monitoring system collects monitoring videos in classrooms of different sizes through one or more cameras deployed in the classroom. Shooting through the camera needs to cover all areas of the entire classroom, and the number of cameras deployed in a larger classroom (such as a lecture hall) needs to be more than that in a smaller classroom (for example, a small class classroom). In addition, corresponding monitoring strategies can also be set according to the size of different classrooms, the specific layout of the classroom, or the total number of people that the classroom can accommodate. For example, in a lecture hall where multiple cameras are deployed, if it is necessary to capture the motion trajectory of a person, and the person may exceed the range that a single camera can shoot when moving, it is necessary to set a monitoring strategy for multiple camera linkage and a corresponding motion trajectory recognition algorithm for the lecture hall; while for a classroom where only one camera is deployed, there is no need to set a monitoring strategy for multiple camera linkage.

[0077] Traditional classroom monitoring equipment usually collects real-time monitoring videos in the classroom (such as videos in H.264 and MPEG-4 formats). In this application, the monitoring videos are converted into video frame images (i.e., monitoring images) for model processing. The daily teaching information recorded in advance by the monitoring system or manually filled in by the teacher is obtained in advance, including: activity records of each class, student attendance, classroom interaction frequency, teaching activity type, etc., which are used to add corresponding activity type labels to each monitoring image.

[0078] After obtaining the surveillance image with the activity type label, feature data is extracted from it and a training data set is constructed. The activity recognition model is trained using the training data set, and the trained activity recognition model is used to detect the target feature data of the target surveillance image collected in real time in the target classroom, identify the activity type and generate output information. The user (teacher) judges the output information to see whether there is a new activity type that has not been identified in the surveillance image. If so, feedback information is provided for incremental training of the activity recognition model to achieve dynamic adjustment of the model.

[0079] In this embodiment, by constructing an activity recognition model based on an adaptive covariance classifier, the adaptability and learning ability of the model to newly emerging activity types are improved. Since the adaptive covariance classifier has the characteristic of dynamically adjusting the decision boundary to adapt to the feature distribution of the new category, the constructed activity recognition model can adjust parameters according to the new activity type, and through incremental training based on a small number of new samples, it can adapt to the dynamic changes of activity types and the emergence of new activities in classroom monitoring images. By obtaining feedback information to dynamically adjust the model, the model can self-learn and continuously optimize based on a small number of samples, thereby improving the generalization ability of the model and meeting the activity recognition needs of the classroom environment.

[0080] In addition, in traditional activity recognition schemes, retraining the model based on newly emerging activity types may reduce the model's recognition effect on old activity types. In this embodiment, the model is not retrained. Instead, the activity recognition model is incrementally trained based on a small number of new samples of the new activity types. This can ensure that the model learns the characteristics of new activity types without forgetting the knowledge of existing activity types, effectively avoiding a decrease in the recognition effect of existing activity types and ensuring the recognition accuracy of the model.

[0081] As an implementation manner of the present application, the activity recognition model includes a first branch and a second branch; wherein the first branch is a neural network based on supervised contrastive learning, and the second branch is an adaptive covariance classifier; using the training data set to train the activity recognition model includes:

[0082] Taking the feature data in the training data set as input data, inputting them into the first branch and the second branch respectively; processing the input data through the first branch to obtain a first feature; and processing the input data through the second branch to obtain a second feature;

[0083] Inputting the first feature and the second feature into a fully connected layer to obtain a recognition result;

[0084] Based on the cross entropy loss function and the contrast loss function, calculating the difference between the recognition result and the label of the input data;

[0085] Based on the difference, a gradient descent algorithm is used to adjust the parameters of the activity recognition model.

[0086] In one embodiment, the activity recognition model includes two branches, namely a first branch constructed based on the SCL (Supervised Contrastive Learning) method and a second branch based on ACC (Adaptive Covariance Classifier). Among them, SCL is a method that uses supervised information to guide the model to learn more discriminative feature representations. The SCL network learns features that can distinguish different categories by comparing the differences between positive samples and negative samples. ACC is a classification method based on the covariance matrix, which dynamically adjusts the decision boundary to adapt to the feature distribution of new categories. Compared with the traditional nearest mean classifier, the ACC network can more accurately describe the feature distribution of new categories, thereby improving the generalization ability of the model.

[0087] Specifically, the first branch of the activity recognition model extracts key visual features from the input data and optimizes the feature representation through contrastive learning to enhance the model's ability to distinguish between different types of activities in the classroom. For example, it enhances the ability to distinguish between raising a hand and writing. These feature representations are trained using a contrastive loss function to quickly generate discriminative recommendations. The first branch can capture complex patterns such as "if there is a hand raising action in the image, then identify it as a question."

[0088] The second branch uses the input data to adjust the covariance matrix through an adaptive mechanism to better match the feature distribution of the new category. This branch consists of adaptive parameters and covariance matrix calculations, which can dynamically adjust the decision boundary to adapt to new activity categories.

[0089] Figure 2 Schematic diagram of an activity recognition model processing feature data in an embodiment of the present application. Figure 2 As shown, the feature data is used as input data and input into the activity recognition model. The first branch network and the second branch network are used to process the data to obtain the first feature and the second feature respectively. The first feature and the second feature are further combined and input into the fully connected layer to obtain the recognition result of the activity type of the input feature.

[0090] Before model training, the training data set is divided into three parts: basic training set, incremental training set and test set. Among them, the basic training set accounts for 70%, and the incremental training set and test set each account for 15%. When training the activity recognition model for the first time, the basic training set and test set are used for iterative training. Each round of iterative training process includes forward propagation, loss calculation, back propagation and model evaluation operations. The incremental training set is used for incremental training of subsequent models. During the model training process, the cross entropy loss function and the contrast loss function are selected to measure the difference between the model's recognition results and the actual activity type label. The gradient descent algorithm (such as the Adam optimizer) is used to train the model, and the model parameters are adjusted to minimize the loss function.

[0091] Optionally, in addition to using the surveillance images in the classroom collected by the monitoring system to construct a training data set, in this embodiment, a general data set can also be used to train the activity recognition model. For example, the ARISC data set (Activity Recognition in School Classroom dataset) is used as the training data. This data set is a continuous learning data set designed for activity recognition in classroom surveillance images. It contains multiple perspectives, multiple activity categories and real-world scenes. The data set has the characteristics of uneven sample distribution and highly similar activity categories, which provides a challenging scenario for the activity recognition task. Using this data set to train the activity recognition model can improve the performance of the model in distinguishing highly similar features and improve the recognition accuracy of activity types.

[0092] By constructing a first branch based on supervised contrastive learning and a second branch based on an adaptive covariance classifier, both the accuracy and robustness of identifying activity types in the classroom are taken into account, so that the model can adapt to different classroom environments and various new activity types on the basis of accurately distinguishing existing activity types.

[0093] As an implementation manner of the present application, incremental training of the activity recognition model based on the new activity type includes:

[0094] Adding a label of the new activity type to the target monitoring image, and adding the target feature data to the training data set;

[0095] The activity recognition model is incrementally trained using the training data set to adjust parameters of the first branch and the second branch.

[0096] In one embodiment, when the model does not recognize a new activity type, the user (e.g., a teacher) sends feedback information to implement dynamic adjustment of the activity recognition model. Specifically, the user submits feedback information (e.g., the new activity type "123" is not recognized). After obtaining the new activity type in the feedback information, a corresponding label (e.g., "123") is added to the target monitoring image, and the target feature data is added to the training data set, and merged with the incremental data set divided from the data set for incremental training of the model. In the case that the model does not recognize a new activity type, the parameters of the first branch and the second branch of the model are adjusted through incremental training to enhance the model's ability to recognize new activity types and to correctly distinguish between new activity types and existing activity types.

[0097] In addition, users can also provide the specific time period of the video involving the monitoring screen of the new activity type according to the pre-set data provision format. The monitoring system extracts the corresponding multiple video frame images based on the time period feedback by the user, and then generates more new sample data for incremental training of the model.

[0098] Figure 3 is a flow chart of training an activity recognition model in one embodiment of the present application. Figure 3 As shown, preprocessing is performed based on the surveillance image to obtain feature data and a training data set is constructed based on the feature data. In the initial stage, the basic training set and the test set are used to perform basic training on the activity recognition model. The trained activity recognition model is used to process the target feature data extracted from the surveillance image of the target classroom and output the recognition result. The user provides feedback information based on the recognition result. The feedback information may be that a new activity type is not recognized, or that the activity type is incorrectly recognized, etc. Based on the feedback information, the activity recognition model is incrementally trained to dynamically adjust the model to adapt to the dynamic changes of activities in the classroom.

[0099] In this embodiment, incremental learning uses the Few-Shot Continual Learning (FSCL) method, which enables the model to continuously learn from new tasks with a small number of samples while minimizing the forgetting of knowledge learned from previous tasks. It is particularly suitable for activity recognition tasks in classroom monitoring images. Through feedback from users (such as teachers) on the activity type recognition results in their classrooms, the model can self-learn and continuously optimize, continuously improve the recognition accuracy and user experience, enhance the flexibility and practicality of the model, and save training costs.

[0100] As an implementation manner of the present application, the feedback information further includes an activity type identification error; after obtaining the feedback information of the output information, it also includes:

[0101] Determine the actual activity type of the target surveillance image and add a corresponding label;

[0102] Adding the target feature data to the training data set;

[0103] The activity recognition model is incrementally trained using the training data set to adjust parameters of the first branch.

[0104] In one embodiment, when the feedback information is an activity type recognition error, the actual activity type of the input data is obtained according to the feedback information submitted by the user (for example, the activity type recognition error should be "456"), and a corresponding label (for example, "456") is added to the target monitoring image. The target feature data is added to the training data set and merged with the incremental data set therein for incremental training of the model. In the case of an existing activity type recognition error, the parameters of the first branch of the model are adjusted through incremental training to enhance the model's ability to correctly distinguish the misidentified activity type from other activity types.

[0105] For example, when monitoring images in a classroom, 10 instances of raising hands to ask questions were identified, but in fact there were 12 instances of raising hands to ask questions (users provided feedback information). Based on the user feedback information, the model parameters of the first branch of the model are dynamically adjusted to enhance the model's recognition ability for the activity type of raising hands to ask questions.

[0106] For example, if multiple teacher feedback models have difficulty identifying student group discussions (a new type of activity), the model parameters are dynamically adjusted, and the first branch and the second branch are gradually adjusted through multi-stage training to enhance the recognition and distinction of group discussion activities.

[0107] In actual applications, the activity type recognition errors and the failure to recognize new activity types in the feedback information usually occur together. When there are multiple feedback information, the model needs to be adjusted step by step, and the branch parameters of each adjustment need to be determined according to the specific feedback information. The following example is used to illustrate.

[0108] For example, in the first recognition, the system identified an average of 10 hand-raising questions per day; and the teacher provided feedback: the model failed to recognize the hand-raising questions of some students. Based on the feedback information, the comparative learning strategy of the first branch of the model is dynamically adjusted to enhance the recognition ability of hand-raising question features, and the new samples fed back by the teacher are added to the incremental data set for incremental learning. During the incremental learning process, the model analyzes whether the recognition errors are regular, such as whether the model always makes inaccurate recognition in a specific time period or a specific activity type.

[0109] In the second recognition, the system recognized 12 times of raising hands to ask questions; the teacher provided feedback: the recognition accuracy of raising hands to ask questions has improved, but it failed to recognize whispering questions (a new activity type). Based on the feedback, further optimization was carried out: the new samples provided by the teacher were added to the incremental data set, the model was incrementally trained, and the model parameters of the first and second branches were adjusted to adapt to the recognition of the activity type of whispering questions.

[0110] The third recognition, the recognition accuracy of the model is further improved.

[0111] In this embodiment, by continuously monitoring, identifying, adjusting and optimizing the model, it is ensured that the system can dynamically adapt to the changing needs of the classroom environment, making activity identification more accurate and in line with actual conditions.

[0112] As an implementation of the present application, a training data set is constructed based on all surveillance images, including:

[0113] Performing data enhancement on each monitoring image to expand the number of monitoring images; the data enhancement includes: random rotation, scaling, cropping, changing brightness or contrast;

[0114] Convert all surveillance images to the same size and color space;

[0115] Extract feature data of all surveillance images as training data, and convert all training data to the same scale;

[0116] Build a training dataset based on all training data.

[0117] In one embodiment, before constructing the training data set, the monitoring images need to be preprocessed, and the process is as follows:

[0118] (1) First, perform data augmentation on surveillance images to expand the amount of data in the training dataset. Specifically, generate new surveillance images by rotating, scaling, cropping, changing color brightness or contrast, etc. to increase the sample diversity of model training;

[0119] (2) Then, considering that the image data collected by different monitoring devices may have differences in format, resolution, and color space, the data format of all monitoring images is standardized and all monitoring images are converted to a unified resolution and color space. For example, all images are converted to RGB color space and the image size is unified to 224×224 pixels;

[0120] (3) Then, the feature data of the surveillance image is extracted using a pre-trained deep learning model to reduce the reliance on the original surveillance image and protect user privacy. In addition, in order to facilitate model processing, the value range of the feature data is standardized, and the feature data is converted to a same scale using, for example, the Min-Max scaling method or the Z-score standardization method. For example, the Min-Max standardization method is used to scale the feature data to between 0 and 1;

[0121] (4) After preprocessing is completed, a training dataset is constructed based on all feature data.

[0122] As an implementation mode of the present application, feature data of all surveillance images are extracted as training data, including:

[0123] Using a pre-trained feature extraction model to extract feature data of all surveillance images as original features; the feature extraction model is constructed based on a neural network;

[0124] Extracting interaction features based on interaction information between people and between people and objects in the surveillance image;

[0125] The original features and the interactive features are used as training data.

[0126] In one embodiment, a pre-trained feature extraction model is used to extract feature data of each monitoring image as original features. The feature extraction model may be a pre-trained model based on deep learning.

[0127] For example, the ResNet50 model, the ViT model, or the CLIP-ViT model, etc. The convolutional layer of the feature extraction model is used to forward propagate the image to obtain the feature representation of the monitoring image.

[0128] In this embodiment, when extracting feature data, in addition to extracting original features based on the information of the objects or people in the surveillance image, it is also necessary to analyze the actions of the people in the surveillance image, identify the interaction information between the people and between the people and the objects in the image, and extract the corresponding interaction features. Specifically, the original features and interaction features of the actions of the people are added to the training data set for the training of the activity recognition model. By extracting interaction features, the model's capture of relational information in the surveillance image is enhanced, and the accuracy of activity type recognition is improved. For example, the model captures the features of the student lowering his head and the features of the student holding a pen, and therefore identifies the activity type as note-taking, rather than misjudging it as the student being absent-minded.

[0129] As an implementation mode of the present application, extracting feature data of all surveillance images also includes:

[0130] Filter the behaviors that are of interest to all current activity types and determine the behaviors whose weights are greater than the weight threshold;

[0131] Based on the screened behaviors, extracting corresponding key behavior features from the surveillance image;

[0132] The key behavior features are added to the training data.

[0133] In one embodiment, the behaviors that are focused on by each activity type to be detected and the weight threshold are determined. For the behaviors that are focused on, the key behavior features of the behaviors that are focused on are extracted from the monitoring image on the basis of extracting the original features. The key behavior features are added to the training data, so that the model can focus on learning the information of the key behaviors during the training process. For example, the behaviors that are focused on by the activity type of speaking in class include: raising hands, standing up, facing the teacher, speaking, etc. Among them, the behavior with the highest weight is raising hands. Therefore, the key behavior features for the behavior of raising hands are extracted from the monitoring image and added to the training data.

[0134] Optionally, in one embodiment, audio can also be extracted from the original surveillance video, and audio features can be further extracted from it as training data. For example, in a campaign scenario of a student class meeting, it is necessary to analyze the student's speeches and discussions, so it is necessary to extract audio data for this time period. In this embodiment, for some special activities, it is necessary to identify them through audio. Therefore, after the surveillance video in the classroom is collected by the monitoring system, in addition to converting the video into frame images, audio is extracted from the video, and audio features are further extracted for training the activity recognition model. The audio features provide auxiliary reference for the recognition results output by the model, thereby improving the accuracy of the model in identifying the activity type.

[0135] As an implementation mode of the present application, extracting feature data of all surveillance images also includes:

[0136] Derived features are calculated based on the original features, and the derived features are added to the training data.

[0137] In one embodiment, for the extracted original features, derived features are further calculated and added to the training data, so that the model can capture the implicit information in the original features by learning the derived features during the training process, thereby improving the accuracy of the model in identifying the activity type. For example, for a student who raises his hand to answer questions more than a threshold number of times during class, the extracted original feature is raising his hand, and further calculation of the derived features can obtain a derived feature that the student has good grades.

[0138] As an implementation manner of the present application, generating output information based on the recognition result includes:

[0139] Obtaining person information and time corresponding to the recognition result;

[0140] Acquiring environmental information of the target classroom;

[0141] Output information is generated based on the recognition result, the person information, the time and the environment information.

[0142] In one embodiment, other monitoring data in the classroom is collected through the classroom monitoring system, and is used to generate output information in combination with the activity recognition results. Specifically, basic information of the monitoring image, character information, classroom environment information, etc. are collected. Among them, the basic information includes: time, image resolution, image format (such as JPEG, PNG), etc.; environmental information includes: classroom number, monitoring equipment deployment location, etc., lighting conditions, indoor layout (such as the placement of tables and chairs), special teaching equipment (such as electronic whiteboards, projectors), etc.; character information includes: number of students, number of teachers and other information.

[0143] After obtaining the recognition result (i.e., activity type) corresponding to the target monitoring image of the target classroom through the activity recognition model, the output information for viewing is finally generated by combining the time, person information, and environmental information corresponding to the target monitoring image. For example, if the recognition result of the target monitoring image is "Raise your hand to ask a question", the corresponding time is 10:00 in the morning, the person involved is student Xiao Ming (student number: A12345), and the environmental information is Class 1, Grade 3 of the high school department, then the output information finally generated is as follows:

[0144] Time: 10:00 am;

[0145] Classroom: Class 1, Grade 3, Senior High School;

[0146] Activity: Students raise their hands to ask questions;

[0147] Students involved: A12345.

[0148] Optionally, after collecting the character information, the character's personal information (such as name or student number) is anonymized or encrypted to protect the user's privacy. For example, a random number is used to replace the user's name or student number, or the name or student number is encrypted to protect the user's privacy.

[0149] Based on the same inventive concept, an embodiment of the present application provides a classroom activity recognition device. Figure 4 , Figure 4 FIG. 1 is a schematic diagram of a classroom activity recognition device 100 proposed in an embodiment of the present application. Figure 4 As shown, the device comprises:

[0150] The preprocessing module 101 is configured to obtain surveillance images of multiple classrooms of different sizes; the surveillance images include: video frame images of different time periods, different viewing angles and different activity types for each classroom; wherein each surveillance image is labeled with a corresponding activity type; extract feature data of the surveillance images, and construct a training data set;

[0151] A training module 102 is configured to train an activity recognition model using the training data set; the activity recognition model is constructed based on an adaptive covariance classifier;

[0152] The recognition module 103 is configured to detect the target feature data using the trained activity recognition model to obtain a recognition result, and generate output information based on the recognition result; the target feature data is feature data generated based on the target monitoring image of the target classroom;

[0153] The feedback module 104 is further configured to obtain feedback information of the output information;

[0154] The training module 102 is further configured to perform incremental training on the activity recognition model based on the new activity type when the feedback information includes an unrecognized new activity type.

[0155] As an implementation manner of the present application, the training module 102 is configured to train the activity recognition model using the training data set, specifically including:

[0156] Taking the feature data in the training data set as input data, inputting them into the first branch and the second branch respectively; processing the input data through the first branch to obtain a first feature; and processing the input data through the second branch to obtain a second feature;

[0157] Inputting the first feature and the second feature into a fully connected layer to obtain a recognition result;

[0158] Based on the cross entropy loss function and the contrast loss function, calculating the difference between the recognition result and the label of the input data;

[0159] Based on the difference, a gradient descent algorithm is used to adjust the parameters of the activity recognition model.

[0160] As an implementation manner of the present application, the training module 102 is configured to perform incremental training on the activity recognition model based on the new activity type, specifically including:

[0161] Adding a label of the new activity type to the target monitoring image, and adding the target feature data to the training data set;

[0162] The activity recognition model is incrementally trained using the training data set to adjust parameters of the first branch and the second branch.

[0163] As an implementation manner of the present application, the feedback information further includes an activity type recognition error; after obtaining the feedback information of the output information, the training module 102 is further configured to perform the following steps:

[0164] Determine the actual activity type of the target surveillance image and add a corresponding label;

[0165] Adding the target feature data to the training data set;

[0166] The activity recognition model is incrementally trained using the training data set to adjust parameters of the first branch.

[0167] As an implementation of the present application, the preprocessing module 101 is configured to construct a training data set based on all surveillance images, including:

[0168] Performing data enhancement on each monitoring image to expand the number of monitoring images; the data enhancement includes: random rotation, scaling, cropping, changing brightness or contrast;

[0169] Convert all surveillance images to the same size and color space;

[0170] Extract feature data of all surveillance images as training data, and convert all training data to the same scale;

[0171] Build a training dataset based on all training data.

[0172] As an implementation manner of the present application, the preprocessing module 101 is configured to extract feature data of all surveillance images as training data, including:

[0173] Using a pre-trained feature extraction model to extract feature data of all surveillance images as original features; the feature extraction model is constructed based on a neural network;

[0174] Extracting interaction features based on interaction information between people and between people and objects in the surveillance image;

[0175] The original features and the interactive features are used as training data.

[0176] As an implementation manner of the present application, the preprocessing module 101 is configured to extract feature data of all surveillance images, and further includes:

[0177] Filter the behaviors that are of interest to all current activity types and determine the behaviors whose weights are greater than the weight threshold;

[0178] Based on the screened behaviors, extracting corresponding key behavior features from the surveillance image;

[0179] The key behavior features are added to the training data.

[0180] As an implementation manner of the present application, the preprocessing module 101 is configured to extract feature data of all surveillance images, and further includes:

[0181] Derived features are calculated based on the original features, and the derived features are added to the training data.

[0182] As an implementation manner of the present application, the recognition module 103 is configured to generate output information based on the recognition result, including:

[0183] Obtaining person information and time corresponding to the recognition result;

[0184] Acquiring environmental information of the target classroom;

[0185] Output information is generated based on the recognition result, the person information, the time and the environment information.

[0186] Based on the same inventive concept, an embodiment of the present application provides a classroom monitoring system, the system comprising:

[0187] The acquisition unit includes: at least one camera deployed in each classroom, used to collect surveillance videos in each classroom; a recording module, used to record the time and environment information corresponding to each surveillance video, and store all surveillance videos in a video database;

[0188] The recognition unit includes: a conversion module and a classroom activity recognition device as provided in the above embodiment; the conversion module is used to convert the monitoring video into a monitoring image;

[0189] A video database, used to store the surveillance videos collected by the collection unit;

[0190] An activity database, used to store description information of each activity, the description information including: a detailed description of the activity, common behaviors and conditions for the activity to occur;

[0191] The pattern database is used to store the identification patterns corresponding to each activity type, and the identification patterns include: feature description, behavior parameters, identification methods and usage scenarios of the activity type.

[0192] Figure 5Schematic diagram of a classroom monitoring system proposed in one embodiment of the present application. Figure 5 As shown, in this embodiment, the system includes one or more cameras deployed in each classroom. The camera continuously monitors the classroom and collects surveillance videos of different time periods and angles in the classroom. The surveillance video contains activity data of people in the classroom, such as students raising their hands, writing, walking, etc. The recording module is used to store the surveillance video collected by the camera in a video database and record the corresponding time and environmental information. The recognition unit converts each surveillance video into a video frame image (surveillance image) through the conversion module, and further inputs it into the classroom activity recognition device for processing to obtain the recognition result of the activity type.

[0193] The system also includes an activity database and a pattern database for storing pre-set activity type descriptions, recognition patterns, etc. In this embodiment, before training the activity recognition model, it is necessary to perform preliminary settings of key parameters such as feature extraction, model parameters, and weights based on the information pre-stored in the activity database and the pattern database.

[0194] Specifically, the activity database stores: detailed descriptions of various classroom activities, common behavioral characteristics (such as raising hands, discussing, using electronic devices), and the conditions under which the activities occur (such as before holidays, during exams). The pattern database stores multiple activity recognition patterns, where each pattern includes the characteristic description of the corresponding activity, behavioral parameters and recognition methods, as well as applicable scenarios, activity types, etc.

[0195] For example, the activity characteristics of absenteeism in class are described as: falling asleep in class, often not looking at the blackboard, and looking down to do one's own things. Behavior parameters may include: looking down for more than 5 minutes, the distance between the head of the person and the desk is less than 10 cm, etc. The method for identifying the activity category of absenteeism in class is through image recognition, and the usage scenario is the class time period.

[0196] It is worth noting that some activities can also be identified through audio recognition, such as distinguishing whether students are laughing and joking or actively discussing learning-related content during a group discussion.

[0197] Based on the same inventive concept, an embodiment of the present application provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the steps in the classroom activity recognition method described in any of the above embodiments of the present application are implemented.

[0198] Based on the same inventive concept, an embodiment of the present application provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the classroom activity recognition method described in any of the above embodiments of the present application are implemented.

[0199] Based on the same inventive concept, an embodiment of the present application provides an electronic device. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the computer program is executed by the processor, the steps in the classroom activity recognition method described in any of the above embodiments of the present application are implemented.

[0200] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0201] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0202] For the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and components involved are not necessarily required by the present application.

[0203] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0204] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0205] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0206] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0207] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic creative concepts. Therefore, the present application is interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0208] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0209] The classroom activity identification method, device and classroom monitoring system provided by the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A classroom activity recognition method, characterized in that: include: Obtain surveillance images of multiple classrooms of different sizes; The monitoring images include: video frame images of different time periods, different viewing angles and different activity types for each classroom; wherein each monitoring image is labeled with a corresponding activity type; Extracting feature data of the monitoring image and constructing a training data set; Using the training data set to train an activity recognition model; the activity recognition model is constructed based on an adaptive covariance classifier; Using the trained activity recognition model to detect target feature data, obtain a recognition result, and generate output information based on the recognition result; the target feature data is feature data generated based on a target monitoring image of a target classroom; Acquire feedback information of the output information, and when the feedback information includes an unrecognized new activity type, perform incremental training on the activity recognition model based on the new activity type.

2. The classroom activity recognition method according to claim 1, characterized in that: The activity recognition model includes a first branch and a second branch; wherein the first branch is a neural network based on supervised contrastive learning, and the second branch is an adaptive covariance classifier; using the training data set to train the activity recognition model includes: Taking the feature data in the training data set as input data, inputting them into the first branch and the second branch respectively; processing the input data through the first branch to obtain a first feature; and processing the input data through the second branch to obtain a second feature; Inputting the first feature and the second feature into a fully connected layer to obtain a recognition result; Based on the cross entropy loss function and the contrast loss function, calculating the difference between the recognition result and the label of the input data; Based on the difference, a gradient descent algorithm is used to adjust the parameters of the activity recognition model.

3. The classroom activity recognition method according to claim 2, characterized in that: Incrementally training the activity recognition model based on the new activity type includes: Adding a label of the new activity type to the target monitoring image, and adding the target feature data to the training data set; The activity recognition model is incrementally trained using the training data set to adjust parameters of the first branch and the second branch.

4. The classroom activity recognition method according to claim 2, characterized in that: The feedback information also includes an activity type identification error; after obtaining the feedback information of the output information, it also includes: Determine the actual activity type of the target surveillance image and add a corresponding label; Adding the target feature data to the training data set; The activity recognition model is incrementally trained using the training data set to adjust parameters of the first branch.

5. The classroom activity recognition method according to claim 1, characterized in that: Build a training dataset based on all surveillance images, including: Performing data enhancement on each monitoring image to expand the number of monitoring images; the data enhancement includes: random rotation, scaling, cropping, changing brightness or contrast; Convert all surveillance images to the same size and color space; Extract feature data of all surveillance images as training data, and convert all training data to the same scale; Build a training dataset based on all training data.

6. The classroom activity recognition method according to claim 5, characterized in that: Extract feature data of all surveillance images as training data, including: Using a pre-trained feature extraction model to extract feature data of all surveillance images as original features; the feature extraction model is constructed based on a neural network; Extracting interaction features based on interaction information between people and between people and objects in the surveillance image; The original features and the interactive features are used as training data.

7. The classroom activity recognition method according to claim 6, characterized in that: Extract feature data of all surveillance images, including: Filter the behaviors that are of interest to all current activity types and determine the behaviors whose weights are greater than the weight threshold; Based on the screened behaviors, extracting corresponding key behavior features from the surveillance image; The key behavior features are added to the training data.

8. The classroom activity recognition method according to claim 6, characterized in that: Extract feature data of all surveillance images, including: Derived features are calculated based on the original features, and the derived features are added to the training data.

9. The classroom activity recognition method according to claim 1, characterized in that: Generating output information based on the recognition result includes: Obtaining person information and time corresponding to the recognition result; Acquiring environmental information of the target classroom; Output information is generated based on the recognition result, the person information, the time and the environment information.

10. A classroom activity recognition device, characterized in that: Used to perform the method according to any one of claims 1 to 9, comprising: A preprocessing module is configured to obtain surveillance images of multiple classrooms of different sizes; the surveillance images include: video frame images of different time periods, different viewing angles and different activity types for each classroom; wherein each surveillance image is labeled with a corresponding activity type; extract feature data of the surveillance images, and construct a training data set; A training module, configured to train an activity recognition model using the training data set; the activity recognition model is constructed based on an adaptive covariance classifier; The recognition module is configured to detect the target feature data using the trained activity recognition model to obtain a recognition result, and generate output information based on the recognition result; the target feature data is feature data generated based on the target monitoring image of the target classroom; A feedback module is further configured to obtain feedback information of the output information; The training module is further configured to, when the feedback information includes an unrecognized new activity type, perform incremental training on the activity recognition model based on the new activity type.

11. A classroom monitoring system, characterized in that: include: The acquisition unit includes: at least one camera deployed in each classroom, used to collect surveillance videos in each classroom; a recording module, used to record the time and environment information corresponding to each surveillance video, and store all surveillance videos in a video database; The recognition unit comprises: a conversion module and the classroom activity recognition device according to claim 10; the conversion module is used to convert the monitoring video into a monitoring image; A video database, used to store the surveillance videos collected by the collection unit; An activity database, used to store description information of each activity, the description information including: a detailed description of the activity, common behaviors and conditions for the activity to occur; The pattern database is used to store the identification patterns corresponding to each activity type, and the identification patterns include: feature description, behavior parameters, identification methods and usage scenarios of the activity type.

Images