A video human behavior recognition method based on high and low resolution dual-mode distillation

Abstract

The application discloses a video human behavior recognition method based on high-low resolution dual-mode distillation. The method comprises the following steps: constructing a deep learning model framework, wherein a first branch comprises a super-resolution module and a super-resolution mode model in sequence, and a second branch comprises a low-resolution mode model; the super-resolution module is used for expanding the resolution of an input video to obtain a super-resolution video; the super-resolution mode model is used for recognizing the human behavior category in the super-resolution video; and the low-resolution mode model is used for recognizing the human behavior category in the input video; a set loss function is used as a supervision signal to train the deep learning model framework; in the training process, the human behavior category output by the super-resolution mode model is used as super-resolution knowledge to guide the training of the low-resolution mode model; and the trained low-resolution mode model is used for human behavior recognition of a target video. The application can effectively improve the human behavior recognition accuracy of a low-resolution video in a real scene.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a video human behavior recognition method based on high and low resolution dual-modal distillation. Background Technology

[0002] Real low-resolution videos are characterized by unclear action details, low information content per frame, high redundancy between adjacent frames, and high noise content. Traditional behavior recognition networks have difficulty accurately recognizing human behavior information in low-resolution videos.

[0003] In existing technologies, mainstream video action recognition models are usually designed for high-resolution models, which perform poorly in low-resolution scenes. Most low-resolution datasets currently used for research are artificially downsampled from high-resolution datasets and do not represent real-world low-resolution videos. The TinyVIRAT dataset fills the gap in real-world low-resolution datasets. Unlike previous artificially downsampled datasets, it contains low-resolution data from real-world scenes, without corresponding high-resolution video data. The challenge of this dataset is that the subjects being identified are filmed at great distances, resulting in low resolution, and there is also interference from lens noise (e.g., Tirupattur P, Rana AJ, Sangam T, et al. TinyAction Challenge: Recognizing Real-world Low-resolution Activities in Videos[J]. arXiv preprint arXiv:2107.11494, 2021).

[0004] In summary, traditional behavior recognition models are designed for high-resolution video data and are not specifically designed for low-resolution data. Therefore, for low-resolution data in real-world scenarios like TinyVIRAT, the accuracy of traditional behavior recognition models is very low. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a video human behavior recognition method based on high- and low-resolution dual-modal distillation. This method includes the following steps:

[0006] A deep learning model framework with two branches is constructed, wherein the first branch contains a super-resolution module and a super-resolution modal model in sequence, and the second branch contains a low-resolution modal model. The super-resolution module is used to reduce noise in the input video, increase motion details, and obtain super-resolution video data. The super-resolution modal model is used to identify the category of the super-resolution video data, and the low-resolution modal model is used to identify the category of the input video.

[0007] The deep learning model framework is trained using a set loss function as a supervision signal. During the training process, the output vector of the super-resolution modality model is used as additional super-resolution knowledge to guide the training of the low-resolution modality model.

[0008] Human behavior recognition is performed on target videos using a trained low-resolution modality model.

[0009] Compared with the prior art, the advantage of this invention is that it introduces a high- and low-resolution dual-modal distillation method, which introduces high-resolution information into the training of the low-resolution video behavior recognition model, thereby improving the accuracy of low-resolution video behavior recognition in real-world scenarios.

[0010] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0012] Figure 1 This is a flowchart of a video human behavior recognition method based on high- and low-resolution dual-modal distillation according to an embodiment of the present invention;

[0013] Figure 2 This is a schematic diagram of a model framework based on high- and low-resolution dual-mode distillation according to an embodiment of the present invention;

[0014] Figure 3 This is a comparison diagram of video data before and after super-resolution according to an embodiment of the present invention. Detailed Implementation

[0015] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0016] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0017] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0018] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0019] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0020] This invention proposes a video-based human behavior recognition method based on high- and low-resolution dual-modal distillation. It employs a model distillation mechanism, using a super-resolution modal model (or super-comparison model) as the teacher model and transferring its knowledge to the student model (i.e., the low-resolution modal model). This allows the student model to flexibly integrate knowledge from the teacher model during training. The super-resolution modal model notices human action details overlooked by the low-resolution modal model, thus serving as a supplement to the low-resolution model's details and guiding its training through model distillation. Integrating knowledge from the high-resolution modal model into the low-resolution modal model improves the model's recognition accuracy.

[0021] See Figure 1 As shown, the provided video human behavior recognition method based on high- and low-resolution dual-modal distillation includes the following steps:

[0022] Step S110: Construct a deep learning model framework with high-resolution branch and low-resolution branch, wherein the high-resolution branch has a super-resolution modal model and the low-resolution branch has a low-resolution modal model.

[0023] See Figure 2 As shown, the proposed deep learning model framework based on high- and low-resolution dual-modal distillation includes two branches, referred to as the high-resolution branch and the low-resolution branch. The high-resolution branch has a super-resolution module (or super-resolution model) and a super-resolution modal model, and the output is the output of the super-resolution modal model; the low-resolution branch has a low-resolution modal model.

[0024] The super-resolution module is used to expand the resolution of the input video, reduce data noise, increase video details, and obtain super-resolution video data. The super-resolution module can adopt various types of structures, as long as it can achieve a good super-resolution video effect.

[0025] The super-resolution modality model is used to identify categories in super-resolution video data, while the low-resolution modality model is used to identify categories in the input video. Various types of deep learning models can be used for both the super-resolution and low-resolution modality models. Experiments show that Ir-CSN and Uniformer models perform well.

[0026] Step S120: Train the deep learning model framework using the set loss function as the supervision signal. During the training process, the output of the super-resolution modality model is used as knowledge and distilled into the low-resolution branch.

[0027] During training, a high- and low-resolution dual-modal model distillation mechanism is adopted, the most important of which are the super-resolution branch, the low-resolution branch, and the interaction between the two branches.

[0028] Specifically, the super-resolution branch mainly includes the following steps:

[0029] Step S11: Input the low-resolution input video into the super-resolution module (such as RealBasicVSR) to obtain super-resolution video data.

[0030] For example, the resolution was expanded from around 70x70 to 224x224.

[0031] In one embodiment, the super-resolution module can be pre-trained, and its parameters are frozen during model distillation training. Introducing the super-resolution module can reduce data noise, while freezing its parameters and performing offline calls can reduce memory consumption during knowledge distillation, save computational resources, and accelerate training.

[0032] Step S12: Use these super-resolution data to train a super-resolution modal model.

[0033] For example, the backbone network of the selected model is the Ir-CSN ResNet152 network.

[0034] Step S13: Input all the training data into the super-resolution model to obtain their corresponding category outputs as knowledge, and then transfer them to the training process of the low-resolution modality model.

[0035] The low-resolution branch mainly includes the following steps:

[0036] Step S21: Input the low-resolution video into the low-resolution modal model.

[0037] The backbone network for low-resolution modal models can also be the Ir-CSN ResNet152 network.

[0038] Step S22: Apply a loss function to the outputs of the low-resolution mode model and the super-resolution mode model. The calculation is expressed as:

[0039]

[0040]

[0041] Where K represents the output of the super-resolution modality model, representing the knowledge of the super-resolution branch. p represents the output of the low-resolution modality model, C represents the number of classes that the dataset needs to distinguish, and α is the weight coefficient of the two loss functions. You can choose the mean squared error (MSE) as shown in equation (2), or you can choose other types of loss functions, such as KLLoss, MSE, etc. The gradient can enable the relevant knowledge of the super-resolution modal model to be integrated into the training of the low-resolution modal model.

[0042] Step S23, transmit back during training. In other words, it is the binary cross-entropy loss function that compares the output p of the low-resolution modality model with the loss of the true label.

[0043] Step S24: The distilled model is tested on the test set, and the recognition result and corresponding F1-Score are output.

[0044] In one embodiment, an online distillation method can be used to train the super-resolution modal model alongside it. This online distillation method does not save the output of the offline super-resolution modal model. During the training of the low-resolution modal model, the low-resolution video is simultaneously input into both the super-resolution module and the super-resolution modal model to obtain the corresponding super-resolution modal model output. This online distillation method requires a large GPU with significant video memory and substantial computing resources, making it difficult to implement.

[0045] Preferably, to reduce memory usage during distillation, an offline distillation method can be used. The super-resolution modality model is pre-trained separately, and all its outputs on the training set are saved. During the training of the deep learning model framework, the offline output files of the super-resolution modality model are used as super-resolution knowledge to guide the learning process of the low-resolution modality model. This design saves the significant memory overhead caused by the mutual distillation between the high-resolution and low-resolution modality models, eliminating the need to import the super-resolution modality model during the training of the low-resolution modality model. This significantly reduces memory usage and greatly shortens training time.

[0046] It should be noted that, in addition to using the dual-mode supervision signal mentioned above, multi-mode supervision can also be used, adding more modal branches, such as adding an optical flow branch, and fusing the optical flow and super-resolution modes into the low-resolution modal model through multiple supervision signals to perform multi-branch model distillation.

[0047] Step S130: Perform video human behavior recognition using the trained low-resolution modality model.

[0048] By using trained low-resolution modal models, human behavior recognition in videos can be performed in various scenarios, especially for low-resolution scenes, which can improve the recognition effect.

[0049] For example, in low-resolution security scenarios, the subject being identified is often far from the camera, and the image is blurry, noisy, and has very low resolution. Such low-resolution recognition scenarios lack corresponding high-resolution video, making it difficult to improve image recognition performance using traditional model training. This invention, however, utilizes a high-low resolution dual-modal model distillation mechanism to fuse knowledge from the super-resolution modal model into the low-resolution raw data, improving the accuracy of behavior recognition in real low-resolution scenarios. This makes automated behavior recognition of blurry low-resolution data possible, reducing blind spots in surveillance cameras.

[0050] Furthermore, the present invention can also be used in the following scenarios:

[0051] 1) In terms of video-assisted refereeing, this invention has certain data augmentation and recognition capabilities for low-resolution scenes in real-world situations. It can assist in judging action categories that are far from the camera, reducing misjudgments caused by the camera being far away or blurry.

[0052] 2) Intelligent Video Montage. Faced with a vast video database, it can automatically categorize low-resolution videos, facilitating search engine searches.

[0053] The model training process involved in this invention can be performed offline on a server or in the cloud. The trained model can then be embedded into an electronic device to achieve real-time image recognition. This electronic device can be a terminal device or a server. Terminal devices include any device such as mobile phones, tablets, personal digital assistants (PDAs), point-of-sale (POS) terminals, in-vehicle computers, and smart wearable devices (smartwatches, virtual reality glasses, virtual reality headsets, etc.). Servers include, but are not limited to, application servers or web servers, and can be independent servers, cluster servers, or cloud servers. For example, in practical model applications, target videos can be captured using surveillance video acquisition terminals, mobile devices, or mobile terminals, uploaded to a cloud server or locally, and then the trained low-resolution modality model can be used to achieve image recognition in various scenarios.

[0054] To further verify the effectiveness of this invention, experiments were conducted. Previous models were not specifically designed for low-resolution real-world scenarios, resulting in low F1-Scores on datasets like TinyVIRAT. This invention introduces a high-low resolution dual-modal distillation method, verifying that after processing with the RealBasicVSR super-resolution model, some important motion details can be reconstructed, and sensor noise is reduced, such as... Figure 3As shown. Experiments have demonstrated that the distillation mechanism employed in this invention improves the recognition F1-Score (an indicator used to measure the accuracy of a classification model, taking into account both precision and recall).

[0055] In summary, compared with the prior art, the present invention has the following technical effects:

[0056] 1) Traditional distillation models for low-resolution scenarios use higher-resolution data corresponding to the low-resolution data. In other words, this low-resolution data is artificially downsampled from the higher-resolution data. However, real-world low-resolution data is directly cropped without downsampling. This low-resolution data lacks corresponding higher-resolution video, increasing the difficulty of recognition. To address the characteristics of real-world low-resolution data—low information content per frame and high noise—this invention redesigns the process by introducing a high-low resolution dual-modal distillation method, improving image recognition performance.

[0057] 2) Due to the high resolution of the input data for the super-resolution modal model, it requires a significant amount of GPU memory. If the super-resolution module is imported simultaneously, and the super-resolution modal model, low-resolution modal model, and other models are trained online together, the GPU memory overhead will be enormous, and the training speed will be extremely slow and inefficient. This invention employs an offline training method for distillation training, effectively avoiding the excessive GPU memory consumption associated with importing and training multiple large models simultaneously, thus accelerating the training speed and reducing computational resource consumption.

[0058] 3) This invention employs a model distillation mechanism for high- and low-resolution dual-modal data, introducing high-resolution information into the training of low-resolution videos, resulting in higher recognition accuracy on datasets such as TinyVIRAT.

[0059] 4) High scalability. This invention can be customized for different modules according to actual needs. For example, the super-resolution module can be selected based on different scenarios through ablation experiments. For other defective data, such as data from nighttime, foggy, noisy, or blurry data, the super-resolution module can be replaced with a module that reduces data defects in the corresponding specific scenario. The information after super-resolution preprocessing can be fused into the training of the original data through distillation. This distillation method can also be extended to other scenarios with defective data.

[0060] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0061] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0062] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0063] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0064] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0065] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0066] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0067] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0068] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A video-based human behavior recognition method based on high- and low-resolution dual-modal distillation, comprising the following steps: A deep learning model framework with two branches is constructed, wherein the first branch contains a super-resolution module and a super-resolution modal model, and the second branch contains a low-resolution modal model. The super-resolution module is used to expand the resolution of the input video to obtain a super-resolution video. The super-resolution modal model is used to identify the human behavior categories in the super-resolution video. The low-resolution modal model is used to identify the human behavior categories in the input video. The deep learning model framework is trained using a set loss function as a supervision signal. During the training process, the human behavior categories output by the super-resolution modality model are used as super-resolution knowledge to guide the training of the low-resolution modality model. Human behavior recognition is performed on target videos using a trained low-resolution modality model; The loss function is set as follows: in, This represents the weighted sum of distillation loss and classification loss. This represents the classification loss between the output class and the true label of the low-resolution modality model. This represents the distillation loss between the output classes of the low-resolution mode model and the output classes of the super-resolution mode model. These are the set weighting coefficients.

2. The method according to claim 1, characterized in that, The classification loss between the output categories of the low-resolution modality model and the output categories of the super-resolution modality model is used. Set to: Where k represents the output of the super-resolution modal model, p represents the super-resolution knowledge vector learned by the super-resolution modal model, C represents the number of categories to be distinguished during the training process, and c represents the category index.

3. The method according to claim 1, characterized in that, The super-resolution module is a pre-trained video super-resolution model. During the training process of the deep learning model framework, the super-resolution module freezes the pre-trained model parameters and generates a super-resolution video corresponding to the input video.

4. The method according to claim 1, characterized in that, The super-resolution modal model is pre-trained offline. During the training process of the deep learning model framework, the obtained super-resolution video is fed into the super-resolution modal model, and a super-resolution knowledge vector is output.

5. The method according to claim 1, characterized in that, During the training process of the deep learning model framework, the input video is input into the low-resolution modal model, and after obtaining the output category vector, the classification loss of the output vector and the true label is calculated. The training process is dynamically supervised by a dual-modal supervision signal, and the classification loss and the supervision signal are backpropagated together for gradient.

6. The method according to claim 1, characterized in that, The super-resolution modal model and the low-resolution modal model are selected from CNN or Transformer models.

7. The method according to claim 1, characterized in that, The deep learning model framework was trained based on the TinyVIRAT dataset.

8. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

9. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

Images