Low-resolution interaction behavior recognition method based on classroom scene

Abstract

The invention requests to protect a low-resolution interaction behavior recognition method based on a classroom scene, and the method comprises the steps: generating a corresponding high-resolution video for a low-resolution video, and achieving the guidance of high-resolution video information for a low-resolution network branch through a high-low dual-resolution knowledge distillation model. According to the knowledge distillation model, an off-line distillation mode is adopted, and self-adaptive weights are added so that low-resolution network branches can learn correct prediction probability distribution. The characteristic distillation loss and the response distillation loss are combined as the distillation loss of the low-resolution network branch, and the distillation loss and the classification loss of the low-resolution network branch form total loss, so that model parameters are trained and updated, and the low-resolution network branch can learn more knowledge. In the reasoning stage, a self-adaptive batch processing normalization layer is added to a low-resolution network branch, so that the learned network can adapt to low-resolution behavior recognition.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing and computer vision, and relates to a low-resolution interactive behavior recognition method based on a classroom scene. Background Art

[0002] With the rapid development of technologies such as intelligent monitoring, virtual reality, and augmented reality, the demand for interactive behavior recognition in low-resolution environments is increasing. For example, in intelligent monitoring systems, due to reasons such as fixed camera positions or limited network bandwidth, the acquired videos may be low-resolution, so it is necessary to recognize and understand human movements and gestures in low-resolution videos. Similarly, in the fields of military security, traffic management, and medical assistance, the recognition of interactive behaviors in low-resolution videos can provide important information support.

[0003] Current research on behavior recognition mainly focuses on high-resolution videos where the actions are clearly visible. However, in real-world safety environments, target activities in videos usually occur at distances with lower resolutions. Low-resolution videos refer to videos of any size below 128×128 pixels, which usually contain less visual information, do not have enough pixels to construct underlying features, and are easily disturbed by noise. Moreover, low-resolution video data naturally has a long-tail distribution, which is also a major challenge for model training.

[0004] Existing low-resolution interactive behavior recognition methods can be roughly divided into two types: one is to capture key spatiotemporal information in low-resolution videos and effectively extract behavior features to improve the accuracy of behavior recognition. The other is to improve the behavior recognition performance of low-resolution videos by transferring knowledge from high-resolution videos to low-resolution video models. In addition, in the Tiny Action Recognition Challenge held at CVPR2022, the TinyVIRAT-v2 dataset was introduced, which contains original low-resolution action videos.

[0005] This paper studies low-resolution interactive behavior recognition in open teaching scenarios, and the research objects are classroom surveillance videos from universities, middle schools, and primary schools across the country. Since student actions in classroom surveillance videos usually occur in the middle and back rows farther away, the image resolution in these areas is low, and the recognition accuracy is also affected. In such low-resolution conditions, identifying students' tiny movements and subtle behavioral changes becomes a major challenge. Therefore, the core difficulty of this study lies in how to effectively extract and identify these subtle and critical behavioral features in low-resolution videos.

[0006] After searching, the application publication number CN116563925A, a face recognition method under low-resolution conditions, includes the following steps: S1, collect and prepare a batch of high-resolution face images and corresponding low-resolution face images, form a high- and low-resolution data set pair, and divide them into a training set and a test set; S2, generate a teacher model by training the training set of high-resolution images, and fix the model parameters. The present invention introduces two linear layers to complete the purpose of mapping the teacher and student features to the same space. After experimental comparison, it is found that the double layer can achieve better results than a single layer or zero layer. Based on the contrast loss of cos similarity, the representation ability of the high-resolution teacher model is distilled into the low-resolution student model, guiding the student model to learn the high-dimensional representation ability of the teacher model, and helping the student model capture the information in the low-resolution image.

[0007] Compare with the patent "A face recognition method under low resolution conditions", patent application number CN116563925A.

[0008] Similarities:

[0009] (1) The data used are high-resolution data and corresponding low-resolution data.

[0010] (2) Both use feature-based knowledge distillation methods.

[0011] Differences:

[0012] (1) The input data of this patent is face image data; while the input of the present invention is classroom video data.

[0013] (2) Based on feature knowledge distillation, the present invention also adds an attention mechanism to select key information areas for knowledge distillation, which avoids redundant information distillation and improves the efficiency of distillation. (3) The present invention also uses a response-based knowledge distillation method and adds adaptive weights to allow the student network to learn more correct classification information.

[0014] (4) The present invention also uses an adaptive batch normalization layer so that during the inference phase, the learned student network can adapt to low-resolution behavior recognition tasks. Summary of the invention

[0015] The present invention aims to solve the above problems of the prior art. A low-resolution interactive behavior recognition method based on classroom scenes is proposed. The technical solution of the present invention is as follows:

[0016] A low-resolution interactive behavior recognition method based on a classroom scene is used to train an interactive behavior recognition model, which includes the following steps:

[0017] Step 1: Generate the corresponding high-resolution video from the low-resolution classroom video through the super-resolution generator RealBasicVSR;

[0018] Step 2: Input the high-resolution video into the high-resolution teacher network for training and save the model parameters; the teacher network is trained with high-resolution input data and can fully capture more details and fine features during the learning process;

[0019] Step 3: The high-resolution teacher network branch loads the pre-trained weights, and the high-resolution video data and low-resolution video data are loaded into the corresponding network branches respectively;

[0020] Step 4: The high-resolution teacher network branch enters the verification mode, and the low-resolution student network branch starts training;

[0021] Step 5: Use the attention mechanism to select key area information in the high-resolution features for distillation and calculate the feature distillation loss L f ;

[0022] Step 6: Calculate the error rate Error of the teacher classification result, perform adaptive response distillation based on Error, and calculate the response distillation loss

[0023] Step 7: Calculate the classification loss L of the low-resolution network branch cls , the overall loss is the characteristic distillation loss L f , Adaptive Response Distillation Loss Classification loss L cls sum;

[0024] Step 8: Perform back propagation based on the overall loss and update the model parameters;

[0025] Step 9: Repeat steps 4-8 until the loss converges;

[0026] Step 10: In the inference phase, the student network enables the adaptive batch normalization layer so that the learned low-resolution student network can adapt to the low-resolution action recognition task.

[0027] Furthermore, the interactive behavior recognition model is an end-to-end behavior recognition model, that is, the interactive behaviors between students and teachers, and between students in the classroom environment are taken as behavior categories to be identified, and the interactive behaviors of characters in low-resolution images in classroom surveillance videos are analyzed through behavior recognition methods.

[0028] Furthermore, the step 1 generates a corresponding high-resolution video from the low-resolution classroom video through a super-resolution generator RealBasicVSR, specifically including:

[0029] The low-resolution classroom video data used is directly captured from the surveillance camera in the classroom. The video source is real-time surveillance images and has not been artificially downsampled or degraded.

[0030] RealBasicVSR is a generative adversarial network model for super-resolution reconstruction. It not only focuses on the clarity of the image during super-resolution reconstruction, but also preserves the natural appearance and texture of the image as much as possible. Equipped with an image pre-cleaning module, RealBasicVSR can effectively aggregate long-term information through propagation, generate more details in fine areas, and thus improve visual quality. In addition, RealBasicVSR can capture dynamic changes and has significant advantages in processing motion and time changes in videos, and can reduce motion blur and lost details.

[0031] Furthermore, the step 2 adopts an offline distillation method to distill the knowledge model, which specifically includes:

[0032] During the training of the knowledge distillation model, the parameters of the teacher network are fixed. The knowledge distillation model transfers the knowledge of the teacher network to the student network, thereby providing more information to help the student network learn. Offline distillation is a common method of knowledge distillation, which specifically refers to transferring the knowledge of a pre-trained teacher model to the student network. Using offline distillation can reduce memory and computing overhead, speed up the training process and shorten training time.

[0033] Furthermore, step 5 uses the attention mechanism to select key area information in the high-resolution features for distillation and calculates the feature distillation loss L f ; Specifically include:

[0034] By introducing the attention mechanism, key areas are selected from high-resolution video features for knowledge transfer, so that the student network can focus on learning key information that is useful for behavior recognition; specifically, the motion attention mechanism is introduced. The motion information is modeled by adjacent frames, focusing on the area that changes between the previous and next frames. The given feature is represented as F r ∈R N ×T×C×H×W , N, T, C, H, W represent the batch size, number of video frames, number of channels, video height and width respectively. The motion feature can be expressed as:

[0035] F m =K*F r [:,t+1,:,:,:]-F r [:,t,:,:,:]

[0036] Where K is a 3*3 two-dimensional convolutional layer;

[0037] The calculation formula of feature distillation loss is:

[0038]

[0039] Among them, F t is the last layer of feature representation extracted by the teacher network, F s It is the last layer of feature representation extracted by the student network.

[0040] Furthermore, the step 6 calculates the error rate Error of the teacher classification result, performs adaptive response distillation based on Error, and calculates the response distillation loss Specifically include:

[0041] The adaptive weight of response distillation is determined by calculating the error rate Error between the teacher network classification result and the true label The adaptive weight formula is as follows:

[0042]

[0043] The calculation formula of response distillation loss is:

[0044]

[0045] Where T and S are the logits vectors of the teacher and student networks.

[0046] Furthermore, the step 7: calculating the low-resolution network branch classification loss L cls , the calculation formula of classification loss is:

[0047] L cls =-[y·log(x)+(1-y)·log(1-x)]

[0048] Where x is the predicted value of the low-resolution network and y is the corresponding true label.

[0049] The overall loss is the characteristic distillation loss L f , adaptive response distillation loss Classification loss L cls The sum of; specifically including;

[0050]

[0051] Furthermore, in the inference phase of step 10, the student network enables an adaptive batch normalization layer so that the learned student network can adapt to low-resolution behavior recognition tasks, specifically including:

[0052] During the inference phase, an adaptive batch normalization module is introduced to improve the migration ability of the learned low-resolution student network on test data, by mixing the statistics of the test video data with the statistics obtained during training, thereby replacing the original parameters of all batch normalization layers in the student network.

[0053] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a low-resolution interactive behavior recognition method based on a classroom scene as described in any one of the items is implemented.

[0054] A non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a low-resolution interactive behavior recognition method based on a classroom scene as described in any one of the items.

[0055] The advantages and beneficial effects of the present invention are as follows:

[0056] The present invention can effectively improve the accuracy of behavior recognition in low-resolution classroom scenarios by combining the attention mechanism with the adaptive response distillation technology. The attention mechanism can help the model identify which areas are more important for behavior recognition, so that it can focus more on these areas during the distillation process. By using the attention mechanism to select key area features in high-resolution videos for distillation, the problem of information loss in low-resolution videos is effectively solved, and the recognition accuracy in low-resolution conditions is improved. And the response distillation loss is dynamically adjusted through adaptive weights, so that the correct classification information in high resolution can be better learned and simulated in low-resolution conditions. In the inference stage, the adaptive batch normalization layer helps to improve the performance of the student network on low-resolution test data sets, and can perform behavior recognition more efficiently and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is an example of an end-to-end behavior recognition model provided by the present invention;

[0058] Figure 2 It is the algorithm flow chart of the present invention. DETAILED DESCRIPTION

[0059] The following will describe the technical solutions in the embodiments of the present invention in detail in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention.

[0060] The technical solution of the present invention to solve the above technical problems is:

[0061] The present invention provides a low-resolution interactive behavior recognition method based on classroom scenarios. Through a distillation learning mechanism, the recognition ability of the student network is gradually optimized, so that it can quickly adapt to and achieve accurate behavior recognition in new classroom scenarios.

[0062] A low-resolution interactive behavior recognition method based on a classroom scene, characterized by comprising the following steps:

[0063] Step 1: Generate the corresponding high-resolution video from the low-resolution classroom video through the super-resolution generator RealBasicVSR;

[0064] Step 2: Input the high-resolution video into the high-resolution teacher network for training and save the model parameters;

[0065] Step 3: Load the pre-trained weights into the high-resolution teacher network branch and load the two types of data into the corresponding network branches;

[0066] Step 4: The high-resolution teacher network branch enters the verification mode, and the low-resolution student network branch starts training;

[0067] Step 5: Use the attention mechanism to select key area information in the high-resolution features for distillation and calculate the feature distillation loss L f .

[0068] Step 6: Calculate the error rate Error of the teacher classification result, perform adaptive response distillation based on Error, and calculate the response distillation loss L KD .

[0069] Step 7: Calculate the classification loss L of the low-resolution network branch cls , the overall loss is the sum of the losses of the three.

[0070] Step 8: Perform backpropagation based on the overall loss and update the model parameters.

[0071] Step 9: Repeat steps 4-8 until the loss converges.

[0072] Step 10: In the inference phase, the student network enables the adaptive batch normalization layer, so that the learned student network can adapt to low-resolution action recognition tasks.

[0073] The high-low dual-resolution knowledge distillation model can select any appropriate end-to-end behavior recognition model according to actual project requirements (detection speed, accuracy, stability, etc.). Figure 1The Ir-CSN behavior recognition model shown can be used in the high-low dual-resolution knowledge distillation model, which can recognize different categories of actions on low-resolution videos. The so-called end-to-end means that the behavior recognition model is generally composed of multiple modules. Different modules implement different functions. It is necessary to compare the results achieved by these modules with the ideal results (labeled) to obtain the error or loss of each module and train the parameters of these modules. However, this training method of superimposing the losses of these modules and updating the parameters of all model parameters by gradient descent is called end-to-end training. Figure 1 For the medium-high-low dual-resolution knowledge distillation model, its module loss includes the feature distillation loss L f , adaptive response distillation loss Classification loss L cls , its total loss is shown in formula (1):

[0074]

[0075] The loss calculated in step 8 is the total loss of the model here.

[0076] The high-low dual-resolution knowledge distillation model trained by the above training method can effectively identify and extract key information that is crucial to low-resolution behavior recognition tasks. This model focuses on high-resolution details in the video and filters out redundant and irrelevant parts, so that the student network can still obtain effective semantic information and spatial features when facing low-resolution inputs, thereby significantly improving the accuracy and robustness of low-resolution behavior recognition. And by enabling the adaptive batch normalization layer, the student network that has undergone knowledge distillation can also adapt well to low-resolution behavior recognition tasks. This model can be applied to open classroom scenarios, has strong scene adaptability, and can also achieve efficient behavior recognition under low-resolution conditions.

[0077] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0078] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0079] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0080] The above embodiments should be understood to be only used to illustrate the present invention and not to limit the protection scope of the present invention. After reading the contents of the present invention, technicians can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A low-resolution interactive behavior recognition method based on classroom scenes, used to train an interactive behavior recognition model, characterized in that: The following steps are involved: Step 1: Generate the corresponding high-resolution video from the low-resolution classroom video through the super-resolution generator RealBasicVSR; Step 2: Input the high-resolution video into the high-resolution teacher network for training and save the model parameters; the teacher network is trained with high-resolution input data and can fully capture more details and fine features during the learning process; Step 3: The high-resolution teacher network branch loads the pre-trained weights, and the high-resolution video data and low-resolution video data are loaded into the corresponding network branches respectively; Step 4: The high-resolution teacher network branch enters the verification mode, and the low-resolution student network branch starts training; Step 5: Use the attention mechanism to select key area information in the high-resolution features for distillation and calculate the feature distillation loss L f ; Step 6: Calculate the error rate Error of the teacher classification result, perform adaptive response distillation based on Error, and calculate the response distillation loss Step 7: Calculate the classification loss L of the low-resolution network branch cls , the overall loss is the characteristic distillation loss L f , adaptive response distillation loss Classification loss L cls sum; Step 8: Perform back propagation based on the overall loss and update the model parameters; Step 9: Repeat steps 4-8 until the loss converges; Step 10: In the inference phase, the student network enables the adaptive batch normalization layer, so that the learned student network can adapt to low-resolution action recognition tasks.

2. According to the low-resolution interactive behavior recognition method based on classroom scenes according to claim 1, it is characterized in that: The interactive behavior recognition model is an end-to-end behavior recognition model, that is, the interactive behaviors between students and teachers, and between students in the classroom environment are taken as the behavior categories to be recognized, and the interactive behaviors of characters in low-resolution images in classroom surveillance videos are analyzed through a behavior recognition method.

3. According to the low-resolution interactive behavior recognition method based on classroom scenes as described in claim 1, it is characterized in that: The step 1 generates a corresponding high-resolution video from the low-resolution classroom video through the super-resolution generator RealBasicVSR, specifically including: The low-resolution classroom video data used is directly captured from the surveillance camera in the classroom. The video source is real-time surveillance images and has not been artificially downsampled or degraded. RealBasicVSR is a generative adversarial network model for super-resolution reconstruction. It not only focuses on the clarity of the image during super-resolution reconstruction, but also preserves the natural appearance and texture of the image as much as possible. Equipped with an image pre-cleaning module, RealBasicVSR can effectively aggregate long-term information through propagation, generate more details in fine areas, and thus improve visual quality. In addition, RealBasicVSR can capture dynamic changes and has significant advantages in processing motion and time changes in videos, and can reduce motion blur and lost details.

4. According to the low-resolution interactive behavior recognition method based on classroom scenes as described in claim 1, it is characterized in that: The step 2 adopts an offline distillation method to distill the knowledge model, which specifically includes: During the training of the knowledge distillation model, the parameters of the teacher network are fixed. The knowledge distillation model transfers the knowledge of the teacher network to the student network, thereby providing more information to help the student network learn. Offline distillation is a common method of knowledge distillation, which specifically refers to transferring the knowledge of a pre-trained teacher model to the student network. Using offline distillation can reduce memory and computing overhead, speed up the training process and shorten training time.

5. According to the low-resolution interactive behavior recognition method based on classroom scenes as described in claim 1, it is characterized in that: Step 5 uses the attention mechanism to select key area information in high-resolution features for distillation and calculates the feature distillation loss L f ; Specifically include: By introducing the attention mechanism, key areas are selected from high-resolution video features for knowledge transfer, so that the student network can focus on learning key information that is useful for behavior recognition; specifically, the motion attention mechanism is introduced. The motion information is modeled by adjacent frames, focusing on the area that changes between the previous and next frames. The given feature is represented as F r ∈R N ×T×C×H×W , N, T, C, H, and W represent the batch size, the number of video frames, the number of channels, and the video height and width, respectively. The motion feature can be expressed as: F m =K*F r [:,t+1,:,:,:]-F r [:,t,:,:,:] Where K is a 3*3 two-dimensional convolutional layer; The calculation formula of feature distillation loss is: where F t is the last layer of feature representation extracted by the teacher network, F s It is the last layer of feature representation extracted by the student network.

6. The low-resolution interactive behavior recognition method based on classroom scenes according to claim 1 is characterized in that: Step 6 calculates the error rate Error of the teacher classification result, performs adaptive response distillation based on Error, and calculates the response distillation loss Specifically include: The adaptive weight of response distillation is determined by calculating the error rate Error between the teacher network classification result and the true label The adaptive weight formula is as follows: The calculation formula of response distillation loss is: Where T and S are the logits vectors of the teacher and student networks.

7. The low-resolution interactive behavior recognition method based on classroom scenes according to claim 1 is characterized in that: Step 7: Calculate the low-resolution network branch classification loss L cls , The calculation formula for classification loss is: L cls =-[y·log(x)+(1-y)·log(1-x)] Where x is the predicted value of the low-resolution network and y is the corresponding true label; The overall loss is the characteristic distillation loss L f , adaptive response distillation loss Classification loss L cls The sum of; specifically including; 8. The low-resolution interactive behavior recognition method based on classroom scenes according to claim 1 is characterized in that: In the inference phase of step 10, the student network enables the adaptive batch normalization layer so that the learned student network can adapt to the low-resolution behavior recognition task, specifically including: During the inference phase, an adaptive batch normalization module is introduced to improve the transferability of the learned student network on test data by mixing the statistics of the test video data with the statistics obtained during training, thereby replacing the original parameters of all batch normalization layers in the student network.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the low-resolution interactive behavior recognition method based on the classroom scene is implemented as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the low-resolution interactive behavior recognition method based on a classroom scene as described in any one of claims 1 to 8 is implemented.

Images