A low-resolution interactive behavior recognition method based on a classroom scene

Abstract

The application claims a low-resolution interactive behavior recognition method based on a classroom scene, which generates corresponding high-resolution videos from low-resolution videos, and uses a high-low dual-resolution knowledge distillation model to realize the guidance of high-resolution video information to the low-resolution network branch. The knowledge distillation model adopts an offline distillation mode, and adds adaptive weights to enable the low-resolution network branch to learn the correct prediction probability distribution. By combining the feature distillation loss and the response distillation loss as the distillation loss of the low-resolution network branch, and combining the classification loss of the low-resolution network branch to form the overall loss, the model parameters are trained and updated, so that the low-resolution network branch can learn more knowledge. In the inference stage, the low-resolution network branch adds an adaptive batch normalization layer, so that the learned network can adapt to low-resolution behavior recognition.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing and computer vision, and relates to a low-resolution interactive behavior recognition method based on a classroom scene. BACKGROUND

[0002] With the rapid development of intelligent monitoring, virtual reality, augmented reality and other technologies, the demand for interactive behavior recognition in low-resolution environments is increasing. For example, in an intelligent monitoring system, due to the fixed position of the camera or the limited network bandwidth, the video obtained may be low-resolution, so it is necessary to recognize and understand human actions, gestures and the like in low-resolution video. Similarly, in the fields of military security, traffic management, medical assistance and the like, recognizing interactive behaviors in low-resolution video can provide important information support.

[0003] Current behavior recognition research mainly focuses on high-resolution videos with clear and visible actions, while in real-world security environments, target activities in videos usually occur at a distance with lower resolution. Low-resolution video refers to any size below 128x128 pixels, which usually contains less visual information, not enough pixels to construct underlying features, and is also susceptible to noise interference. Moreover, natural low-resolution video data follows a long-tail distribution, which is also a great challenge for model training.

[0004] Existing low-resolution interactive behavior recognition methods can be roughly divided into two categories: one is to capture key spatio-temporal information in low-resolution video and effectively extract behavior features to improve the accuracy of behavior recognition. The other is to transfer the knowledge of high-resolution video to the low-resolution video model to improve the behavior recognition performance of low-resolution video. In addition, in the Micro Action Recognition Challenge held in CVPR 2022, the TinyVIRAT-v2 dataset was introduced, which contains original low-resolution action videos.

[0005] The present application studies low-resolution interactive behavior recognition in an open teaching scene, and the research object is classroom monitoring video from primary and secondary schools across the country. Since the student actions in the classroom monitoring video usually occur in the far back row seats, the image resolution of these areas is low, and the recognition accuracy is also affected. In this low-resolution case, recognizing the micro actions and subtle behavior changes of students becomes a great challenge. Therefore, the core difficulty of the present research is how to effectively extract and recognize these subtle and key behavior features in low-resolution video.

[0006] After searching, the application publication number CN116563925A, a low-resolution face recognition method, includes the following steps: S1, collect and prepare a batch of high-resolution face images and corresponding low-resolution face images, form high-low data set pairs, and divide them into training set and test set; S2, generate a teacher model by training the training set of high-resolution images, and fix the model parameters. The invention introduces two linear layers to achieve the purpose of mapping teacher and student features to the same space. Through experimental comparison, it is found that double-layer is better than single-layer or zero-layer in achieving better results. The contrast loss based on cos similarity distills the representation ability of the high-resolution teacher model into the low-resolution student model, guides the student model to learn the high-dimensional representation ability of the teacher model, and helps the student model to capture information in low-resolution pictures.

[0007] Compared with the patent "a low-resolution face recognition method", the patent application number is CN116563925A.

[0008] Same points:

[0009] (1) The data used are high-resolution data and corresponding low-resolution data.

[0010] (2) Both use feature-based knowledge distillation method.

[0011] Difference:

[0012] (1) The data input in this patent is face image data; while the input of this invention is classroom video data.

[0013] (2) This invention adds attention mechanism to select key information area for knowledge distillation based on feature knowledge distillation, which avoids redundant information distillation and improves the efficiency of distillation.(3) This invention also uses response-based knowledge distillation method and adds adaptive weight to let the student network learn more correct classification information.

[0014] (4) This invention also uses adaptive batch normalization layer, so that the learned student network can adapt to low-resolution behavior recognition task in inference stage. Invention content

[0015] This invention aims to solve the problems of the above prior art. A low-resolution interactive behavior recognition method based on classroom scene is proposed. The technical solution of this invention is as follows:

[0016] A low-resolution interactive behavior recognition method based on classroom scene is used to train interactive behavior recognition model, which includes the following steps:

[0017] Step 1: Generate a corresponding high-resolution video for the low-resolution classroom video through the super-resolution generator RealBasicVSR;

[0018] Step 2: Input the high-resolution video into the high-resolution teacher network for training, and save the model parameters; the teacher network uses high-resolution input data for training, and can fully capture more details and fine features during the learning process;

[0019] Step 3: Load the pre-trained weights into the high-resolution teacher network branch, and load the high-resolution video data and low-resolution video data into the corresponding network branches;

[0020] Step 4: The high-resolution teacher network branch enters the verification mode, and the low-resolution student network branch starts training;

[0021] Step 5: Use the attention mechanism to select key area information in the high-resolution feature for distillation, and calculate the feature distillation loss L f ;

[0022] Step 6: Calculate the error rate Error of the teacher classification result, and perform adaptive response distillation according to Error, and calculate the response distillation loss

[0023] Step 7: Calculate the low-resolution network branch classification loss L cls , and the overall loss is the sum of the feature distillation loss L f , the adaptive response distillation loss and the classification loss L cls ;

[0024] Step 8: Perform back propagation according to the overall loss to update the model parameters;

[0025] Step 9: Repeat steps 4-8 until the loss converges;

[0026] Step 10: Inference stage, enable adaptive batch normalization layer for the student network, so that the learned low-resolution student network can adapt to the low-resolution behavior recognition task.

[0027] Further, the interactive behavior recognition model is an end-to-end behavior recognition model, that is, the interactive behavior between students and teachers, and students in the classroom environment is taken as a behavior class to be recognized, and the interactive behavior of the characters in the low-resolution image in the classroom monitoring video is analyzed through a behavior recognition method.

[0028] Further, the step 1 generates a corresponding high-resolution video for the low-resolution classroom video through the super-resolution generator RealBasicVSR, specifically including:

[0029] The low-resolution classroom video data used is directly intercepted from a monitoring camera in the classroom, and the video source is real-time monitoring image, without any human downsampling or degradation processing;

[0030] RealBasicVSR is a generative adversarial network model for super-resolution reconstruction, which not only focuses on the clarity of the image, but also tries to preserve the natural appearance and texture of the image as much as possible. With the image pre-cleaning module, RealBasicVSR can effectively aggregate long-term information through propagation, generate more details in fine areas, and improve visual quality. Moreover, RealBasicVSR can capture dynamic changes, which has a significant advantage in handling motion and temporal changes in videos, reducing motion blur and lost details.

[0031] Further, the step 2 adopts an offline distillation manner for the knowledge distillation model, specifically including:

[0032] During the training process of the knowledge distillation model, the parameters of the teacher network are fixed. The knowledge distillation model transfers its knowledge to the student network through the teacher network, thereby providing more information to help the student network learn. Offline distillation is a common way of knowledge distillation, specifically referring to transferring the knowledge of a pre-trained teacher model to the student network. By adopting the offline distillation manner, the memory and computational overhead can be reduced, accelerating the training process and shortening the training time.

[0033] Further, the step 5 uses an attention mechanism to select key region information in the high-resolution feature for distillation, and calculates the feature distillation loss L f ; specifically including:

[0034] By introducing an attention mechanism, key regions are selected from high-resolution video features for knowledge transfer, so that the student network can focus on learning key information useful for behavior recognition. Specifically, a motion attention mechanism is introduced, and motion information is modeled by adjacent frames, focusing on the regions that change between the previous and subsequent frames. The given feature representation is F r ∈R N ×T×C×H×W , N, T, C, H, and W represent batch size, video frame number, channel number, video height, and width, respectively. The motion feature can be represented as:

[0035] F m =K*F r [:,t+1,:,:,:]-F r [:,t,:,:,:]

[0036] where K is a 3*3 two-dimensional convolution layer;

[0037] The formula for calculating the feature distillation loss is:

[0038]

[0039] Where F t is the last layer feature representation extracted by the teacher network, F s is the last layer feature representation extracted by the student network.

[0040] Further, the step 6 calculates the error rate Error of the teacher classification result, and performs adaptive response distillation according to Error to calculate the response distillation loss Specifically, it includes:

[0041] The adaptive weight of response distillation is determined by calculating the error rate Error between the teacher network classification result and the real label The adaptive weight formula is as follows:

[0042]

[0043] The formula for calculating the response distillation loss is:

[0044]

[0045] Where T and S are logits vectors of the teacher and student networks.

[0046] Further, the step 7: calculate the low-resolution network branch classification loss L cls The formula for calculating the classification loss is:

[0047] L cls = -[y·log(x)+(1-y)·log(1-x)]

[0048] Where x is the predicted value of the low-resolution network, and y is the corresponding real label.

[0049] The overall loss is the sum of the feature distillation loss L f , the adaptive response distillation loss and the classification loss L cls ; Specifically, it includes:

[0050]

[0051] Further, the step 10 inference stage, the student network enables adaptive batch normalization layer, so that the learned student network can adapt to the low-resolution behavior recognition task, specifically including:

[0052] In the inference stage, an adaptive batch normalization module is introduced to improve the migration ability of the learned low-resolution student network on the test data, and by mixing the statistical information of the test video data with the statistical information obtained during training, the original parameters of all batch normalization layers in the student network are replaced.

[0053] An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, which, when executed by the processor, implements the low-resolution interactive behavior recognition method based on a classroom scene according to any one of the preceding embodiments.

[0054] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the low-resolution interactive behavior recognition method based on a classroom scene according to any one of the preceding embodiments.

[0055] The advantages and beneficial effects of the present application are as follows:

[0056] The present application can effectively improve the behavior recognition accuracy in a low-resolution classroom scene by combining attention mechanisms and adaptive response distillation techniques. Attention mechanisms can help the model identify which areas are more important for behavior recognition, so that more focus is placed on these areas during the distillation process. By using attention mechanisms to select key region features in high-resolution videos for distillation, the problem of information loss in low-resolution videos is effectively addressed, improving recognition accuracy in low-resolution situations. And by dynamically adjusting the response distillation loss with adaptive weights, the correct classification information in high-resolution can be better learned and simulated in low-resolution situations. In the inference stage, adaptive batch normalization layers help improve the performance of student networks on low-resolution test datasets, enabling more efficient and accurate behavior recognition. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 is an example of an end-to-end behavior recognition model provided by the present application;

[0058] Figure 2 is an algorithm flowchart of the present application. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be described in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments of the present application.

[0060] The technical solution of the present application to solve the above technical problems is:

[0061] The application provides a low-resolution interactive behavior recognition method based on a classroom scene, which gradually optimizes the recognition ability of a student network through a distillation learning mechanism, so as to quickly adapt to a new classroom scene and realize accurate behavior recognition.

[0062] A low-resolution interactive behavior recognition method based on a classroom scene, characterized by comprising the following steps:

[0063] Step 1: generating a corresponding high-resolution video through a super-resolution generator RealBasicVSR for a low-resolution classroom video;

[0064] Step 2: inputting the high-resolution video into a high-resolution teacher network for training and saving model parameters;

[0065] Step 3: loading pre-training weights for the high-resolution teacher network branch, and loading two kinds of data into corresponding network branches;

[0066] Step 4: the high-resolution teacher network branch enters a verification mode, and the low-resolution student network branch starts training;

[0067] Step 5: using an attention mechanism to select key area information in the high-resolution feature for distillation, and calculating a feature distillation loss L f .

[0068] Step 6: calculating an error rate Error of the teacher classification result, and performing adaptive response distillation according to the Error, and calculating a response distillation loss L KD .

[0069] Step 7: calculating a low-resolution network branch classification loss L cls , and the overall loss is the sum of the three losses.

[0070] Step 8: updating the model parameters according to the overall loss through back propagation.

[0071] Step 9: repeating steps 4-8 until the loss converges.

[0072] Step 10: in the inference stage, the student network enables an adaptive batch normalization layer, so that the learned student network can adapt to the low-resolution behavior recognition task.

[0073] The high-low dual-resolution knowledge distillation model can select any suitable end-to-end behavior recognition model according to actual project requirements (detection speed, accuracy, stability, etc.). For example, Figure 1The Ir-CSN behavior recognition model shown can be used for high- and low-resolution dual-resolution knowledge distillation models, enabling it to identify different categories of actions in low-resolution videos. End-to-end training refers to the fact that behavior recognition models typically consist of multiple modules, each performing a different function. The results from these modules are compared with the ideal result (labeled) to determine the error, or loss, of each module, and the parameters of these modules are then trained. However, end-to-end training involves summing the losses from these modules and updating the parameters of the model using gradient descent. Figure 1 For a dual-resolution knowledge distillation model with medium, high, and low resolutions, its module loss includes the feature distillation loss L. f Adaptive response distillation loss Classification loss L cls Its total loss is shown in formula (1):

[0074]

[0075] The loss calculated in step 8 is the total loss of the model here.

[0076] The high- and low-resolution dual-resolution knowledge distillation model trained using the above methods can effectively identify and extract key information crucial for low-resolution behavior recognition tasks. By focusing on high-resolution details in the video and filtering out redundant and irrelevant parts, the model enables the student network to still acquire effective semantic information and spatial features when faced with low-resolution input, thus significantly improving the accuracy and robustness of low-resolution behavior recognition. Furthermore, by enabling an adaptive batch normalization layer, the knowledge-distilled student network can also adapt well to low-resolution behavior recognition tasks. This model can be applied to open classroom scenarios, exhibiting strong scene adaptability and achieving efficient behavior recognition under low-resolution conditions.

[0077] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0078] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0079] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article or apparatus that includes a list of elements not only includes those elements, but also includes other elements not explicitly listed, or inherent to such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0080] The above embodiments should be understood as merely illustrative of the present application and not restrictive of the scope of protection of the present application. After reading the description of the present application, those skilled in the art can make various modifications or changes to the present application, and these equivalent changes and modifications also fall within the scope defined by the claims of the present application.

Claims

1. A low-resolution interactive behavior recognition method based on a classroom scene, used for training an interactive behavior recognition model, characterized in that, Comprise the following steps: Step 1: generate corresponding high-resolution video for low-resolution classroom video through super-resolution generator RealBasicVSR; Step 2: input high-resolution video into high-resolution teacher network for training, save model parameters; the teacher network uses high-resolution input data for training, and can fully capture more details and fine features in the learning process; Step 3: load the pre-training weight of the high-resolution teacher network branch, and load the high-resolution video data and low-resolution video data into the corresponding network branch respectively; Step 4: the high-resolution teacher network branch enters the verification mode, and the low-resolution student network branch starts training; Step 5: Select key region information in high-resolution features using attention mechanism for distillation, calculate feature distillation loss L f ; Step 6: Calculate the error rate Error of the teacher classification result, and perform adaptive response distillation according to Error to calculate the response distillation loss Step 7: Compute low resolution network branch classification loss L cls , the overall loss is the feature distillation loss L f , the adaptive response distillation loss classification loss L cls ; Step 8: according to the total loss, the model parameters are updated by back propagation; Step 9: repeat steps 4-8 until the loss converges; Step 10: inference stage, student network enables adaptive batch normalization layer, so that the learned student network can adapt to the low-resolution behavior recognition task.

2. The low-resolution interaction behavior recognition method based on a classroom scene according to claim 1, characterized in that, The interactive behavior recognition model is an end-to-end behavior recognition model, that is, the interactive behavior between students and teachers, students and students in the classroom environment is taken as the behavior class to be recognized, and the interactive behavior of the characters in the low-resolution image in the classroom monitoring video is analyzed by the behavior recognition method.

3. The low-resolution interaction behavior recognition method based on a classroom scene according to claim 1, characterized in that, The step 1 generates corresponding high-resolution video for low-resolution classroom video through super-resolution generator RealBasicVSR, specifically comprising: The low-resolution classroom video data used is directly intercepted from the monitoring camera in the classroom, and the video source is real-time monitoring image, which has not been subjected to any human downsampling or degradation processing; RealBasicVSR is a generative adversarial network model for super-resolution reconstruction, which not only focuses on the clarity of the image, but also tries to preserve the natural appearance and texture of the image as much as possible; by equipping an image pre-cleaning module, RealBasicVSR can effectively aggregate long-term information through propagation, generate more details in fine areas, and improve visual quality; and RealBasicVSR can capture dynamic changes, has a significant advantage in processing motion and temporal changes in video, and can reduce motion blur and lost details.

4. The low-resolution interaction behavior recognition method based on a classroom scene according to claim 1, characterized in that, The step 2 adopts offline distillation for the knowledge distillation model, specifically comprising: During the training process of the knowledge distillation model, the parameters of the teacher network are fixed; the knowledge distillation model transfers its knowledge to the student network through the teacher network, thereby providing more information to help the student network learn.

5. The low-resolution interaction behavior recognition method based on a classroom scene according to claim 1, characterized in that, The step 5 uses an attention mechanism to select key area information in high-resolution features for distillation, and calculates a feature distillation loss L f ; specifically comprising: By introducing an attention mechanism, the student network can focus on learning key information useful for behavior recognition by screening key areas from high-resolution video features for knowledge transfer. Specifically, a motion attention mechanism is introduced, motion information is modeled by adjacent frames, and the area of change between the previous and subsequent frames is focused on. Given the feature representation F r ∈R N ×T×C×H×W N, T, C, H, and W represent batch size, video frame number, channel number, video height, and width, respectively, and the motion feature can be represented as: F m = K * F r [:, t + 1, :, :, :] - F r [:, t, :, :, :] K is a 3*3 two-dimensional convolution layer; The calculation formula of feature distillation loss is: where F t is the last layer feature representation extracted by the teacher network, F s is the last layer feature representation extracted by the student network.

6. The low-resolution interaction behavior recognition method based on a classroom scene according to claim 1, characterized in that, The step 6 calculates the error rate Error of the teacher classification result, and performs adaptive response distillation according to the Error to calculate a response distillation loss Specifically comprises: The adaptive weight responding to distillation is determined by calculating the error rate Error between the classification result of the teacher network and the real label The adaptive weight formula is as follows: The calculation formula of response distillation loss is: Where T and S are logits vectors of teacher and student networks.

7. The low-resolution interaction behavior recognition method based on a classroom scene according to claim 1, characterized in that, The step 7: calculating low-resolution network branch classification loss L cls , The calculation formula of classification loss is: L cls = -[y log(x) + (1 - y) log(1 - x)] Where x is the predicted value of the low-resolution network, and y is the corresponding true label; The overall loss is a feature distillation loss L f , an adaptive response distillation loss The classification loss L cls , in particular comprises; 8. The low-resolution interaction behavior recognition method based on a classroom scene according to claim 1, characterized in that, The step 10 inference stage, student network enables adaptive batch normalization layer, so that the learned student network can adapt to the low-resolution behavior recognition task, specifically comprising: In the inference stage, an adaptive batch normalization module is introduced to improve the transferability of the learned student network on the test data, and by mixing the statistical information of the test video data with the statistical information obtained during training, the original parameters of all batch normalization layers in the student network are replaced.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor implements the low-resolution interaction behavior recognition method based on a classroom scene according to any one of claims 1-8 when executing the program.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the low-resolution interaction behavior recognition method based on a classroom scene according to any one of claims 1-8 when executed by the processor.

Images