Single-stage low-resolution action recognition method and system based on super-resolution enhancement

Abstract

The application discloses a single-stage low-resolution action recognition method and system based on super-resolution enhancement, establishes an end-to-end unified single-stage low-resolution video action recognition framework, and is used for realizing action recognition; the framework integrates a super-resolution module and an action recognition module, is oriented to action recognition in a single stage, only needs low-resolution data and does not depend on high-resolution data, and jointly trains the super-resolution and action classification; from the perspective of no-reference image quality evaluation, spatial consistency is maintained by evaluating the change amplitude of two kinds of features of the super-resolution module input and output on self-consistency and neighborhood consistency; a comprehensive loss function composed of a self-consistency loss, a neighborhood consistency loss and a cross-entropy loss is set, and end-to-end optimization is realized by backward propagation in the whole framework. The application jointly optimizes video super-resolution and action classification in a single stage, and can learn more robust and comprehensive action features.

Description

Technical Field

[0001] This invention relates to the field of computer vision human motion recognition technology, and more specifically to a single-stage low-resolution motion recognition method, system, device, and medium based on super-resolution enhancement. Background Technology

[0002] With the development of mobile internet and smartphones, video cameras are becoming increasingly ubiquitous. Millions of cameras record people's daily activities, causing serious privacy issues. We hope that the development of video technology and the data explosion can help us understand and assist human production and life activities, but at the same time, protecting personal privacy is also crucial. Capturing, transmitting, and storing video at extremely low resolution is considered a simple and effective solution. Low-resolution (LR) video is widespread in the real world and can be easily accessed through various mobile devices, greatly reducing the cost of video recording and storage burden.

[0003] This presents a new challenge for computer vision—how to recognize human activity in low-resolution videos. While existing video action recognition technologies have made remarkable progress, these methods rely on the acquisition of sufficiently rich semantic and depth information in the collected data. On tasks based on low-resolution data, the effectiveness of action recognition methods is severely diminished. Therefore, exploring an effective action recognition method that performs well in low-resolution videos is a task of great significance.

[0004] Many deep learning-based methods have been developed to address this task. Currently popular action recognition methods based on convolutional neural networks, such as two-stream networks, are not directly applicable to low spatial resolution videos. Thanks to the development of super-resolution (SR) technology, the idea of ​​learning to extract features from super-resolution videos has been widely applied in various research fields, such as face recognition, medical imaging, object detection, satellite imaging, and high-definition display imaging, achieving very good results. In the field of action recognition, many scholars have also explored super-resolution-based research, mainly in the following two areas:

[0005] (1) Action recognition methods that directly target low-resolution raw data. These methods do not enhance the resolution of the video itself from the perspective of the data, thus placing higher demands on the model's feature extraction and fusion capabilities. Due to the difficulty in extracting high-level semantic features from low-resolution data, many studies have used paired high-resolution data to train networks.

[0006] (2) Low-resolution action recognition methods based on super-resolution enhancement. These methods often perform low-resolution action recognition in two stages: first, super-resolution enhancement is performed on the low-resolution video, and then the super-resolution result is used to perform action recognition. This two-stage framework separates the super-resolution enhancement module and the action classification module, training them on different datasets respectively, making this framework approximate an action recognition model with an added data preprocessing step.

[0007] The low resolution characteristic limits the effective acquisition of spatial domain information, making it difficult for action recognition methods that directly target raw data to extract robust low-resolution features. Furthermore, research on super-resolution enhancement is geared towards action recognition tasks, not visual quality. Whether visual quality determines the quality of visual analysis results, such as the accuracy of action recognition, remains an open question. Since video analysis is performed by computers rather than humans, SR methods optimized for visual quality may not be suitable for action recognition tasks. Studies have found that super-resolution enhancement methods often enhance interpolated frames with image details; these details can be beneficial or detrimental to action recognition, depending on the region where the details are added. Therefore, separating super-resolution and action recognition in a two-stage model may lead to suboptimal recognition performance. Moreover, in new real-world application scenarios, only low-resolution raw data is often available, significantly limiting the practical application capabilities of such models.

[0008] In super-resolution research, the nonlocal similarity prior assumption—that small patches tend to repeat within the same image—has proven to be a strong prior for natural images. Nonlocal attention modules significantly improve super-resolution performance by exploring nonlocal self-similarity features and embedding nonlocal modules into deep networks. However, global nonlocal attention requires calculating the mutual similarity of features across all pixel locations, resulting in a very high computational cost, quadratic to the image size. Therefore, enabling super-resolution networks to possess nonlocal modeling capabilities while keeping computational overhead within an acceptable range is another important challenge.

[0009] In conclusion, developing an action recognition model that learns downstream low-resolution human behavioral features in an end-to-end manner has significant research and application value. Summary of the Invention

[0010] To address the shortcomings of the existing technologies, this invention provides a single-stage low-resolution action recognition scheme based on super-resolution enhancement, which addresses the three major challenges of existing computer vision human action recognition technologies: weak practical application capability, high computational complexity, and high-precision action recognition.

[0011] To achieve the above objectives, this invention proposes a single-stage low-resolution action recognition method based on super-resolution enhancement. An end-to-end unified single-stage low-resolution video action recognition framework is established for action recognition. This framework integrates a super-resolution module and an action recognition module, focusing on action recognition within a single stage. It requires only low-resolution data and does not rely on high-resolution data, jointly training the super-resolution and action classification modules. From the perspective of no-reference image quality evaluation, spatial consistency is maintained by assessing the variation magnitudes of the input and output features of the super-resolution module in terms of self-consistency and neighborhood consistency. A comprehensive loss function consisting of a linear combination of self-consistency loss, neighborhood consistency loss, and cross-entropy loss is set, and end-to-end optimization is achieved through backpropagation throughout the framework.

[0012] Furthermore, in the super-resolution module, a lightweight residual block is set for feature extraction, and a non-local attention module that combines sparsity is embedded into the main body of the super-resolution module.

[0013] Furthermore, the lightweight residual block contains two Shuffle blocks. A ReLU activation layer is set between the first and second Shuffle blocks. After the second Shuffle block, the input of the lightweight residual block is added element-wise to achieve the residual structure.

[0014] Furthermore, the method of embedding a non-local attention module that incorporates sparsity into the super-resolution module body is achieved by inserting non-local sparse attention blocks between lightweight residual blocks, thereby introducing sparse attention to prevent the network from concentrating on noisy and less information-rich locations.

[0015] Furthermore, a nonlocal sparse attention block is inserted between every 8 lightweight residual blocks.

[0016] Moreover, when implementing action recognition based on the end-to-end unified single-stage low-resolution video action recognition framework, low-resolution action video data is used as input. First, the super-resolution feature is obtained through the super-resolution module, then it is input into the action classifier to output the action category, and then the comprehensive loss function is calculated to jointly optimize the entire framework.

[0017] On the other hand, the present invention provides a computer device / apparatus / system, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the single-stage low-resolution action recognition method based on super-resolution enhancement as described above.

[0018] On the other hand, the present invention provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the single-stage low-resolution action recognition method based on super-resolution enhancement as described above.

[0019] On the other hand, the present invention provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the steps of the single-stage low-resolution action recognition method based on super-resolution enhancement as described above.

[0020] This invention designs an end-to-end unified single-stage low-resolution video action recognition framework. In a single stage, it jointly optimizes video super-resolution and action classification to learn more robust and comprehensive action features. The explored model cascades super-resolution enhancement and action classification into a single-stream framework, significantly enhancing low-resolution video frames to facilitate action classification. To better integrate the super-resolution and action recognition modules, a comprehensive loss function is constructed to maintain the smoothness of the network structure. With the help of this loss, the network directly learns from the low-resolution video and captures optimized features for action recognition. This framework is highly flexible, allowing integration with any super-resolution module and action recognition network. Furthermore, a lightweight and efficient super-resolution module is improved based on the MDSR model, enhancing the sparsity of nonlocal operations through a nonlocal sparse attention module. The sparsity constraint allows the module to maintain global modeling capabilities while focusing only on highly relevant and information-rich local regions, thereby improving the robustness and efficiency of the super-resolution module.

[0021] The present invention is simple and convenient to implement, highly practical, and solves the problems of low practicality and inconvenience in actual application of related technologies. It can improve user experience and has significant market value. It also has important scientific research value and practical engineering significance in areas such as human behavior analysis and security monitoring. Attached Figure Description

[0022] Figure 1 This is a framework structure diagram of an embodiment of the present invention.

[0023] Figure 2 This is a structural diagram of the improved super-resolution module used in the embodiments of the present invention. Detailed Implementation

[0024] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] This invention provides a single-stage low-resolution action recognition method based on super-resolution enhancement. The main approach is to establish an end-to-end unified single-stage low-resolution video action recognition framework, upon which action recognition is achieved.

[0026] 1) Integrate the super-resolution module and the action recognition module into a single framework. In a single stage, the action recognition is the guiding principle, requiring only low-resolution data and not relying on high-resolution data. The super-resolution and action classification are trained together.

[0027] 2) From the perspective of no-reference image quality evaluation, spatial consistency is maintained by evaluating the magnitude of changes in self-consistency and neighborhood consistency of the input and output features of the super-resolution module.

[0028] 3) Set up a comprehensive loss function consisting of a linear combination of self-consistency loss, neighborhood consistency loss, and cross-entropy loss, and achieve end-to-end optimization through backpropagation throughout the framework.

[0029] Furthermore, embodiments of the present invention provide an improved lightweight and efficient super-resolution module, including the following specific improvements:

[0030] 1) Design lightweight residual blocks for feature extraction. The lightweight residual block uses two shuffle blocks instead of expensive convolution operations. A ReLU activation layer is placed between the two shuffle blocks, and finally, element-wise addition with the input is performed to achieve the residual structure. That is, a ReLU activation layer is placed between the first and second shuffle blocks, and after the second shuffle block, element-wise addition with the input of the lightweight residual block is performed to achieve the residual structure. The specific structure of the shuffle block can adopt existing technologies. A preferred implementation is to divide the input into two branches through channel separation: one branch performs an identity mapping, and the other undergoes convolution. The two branches are connected at the end, and finally, channel shuffling is used to achieve inter-channel information exchange. For specific implementation details, the shuffle block structure can refer to relevant literature, such as ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design. in 15th European Conference on Computer Vision (ECCV). 2018. Munich, Germany, 2018, pp. 122-138.

[0031] 2) Embed a non-local attention module with sparsity into the main body of the super-resolution module. The specific structure of the non-local sparse attention module can adopt existing techniques, dividing the deep spatial features into different groups through locality-sensitive hashing, and calculating attention only within each group to capture the elements most relevant to the query globally, thus limiting computational costs to an acceptable range. For specific implementation details, refer to relevant literature, such as Y. Mei, Y. Fan and Y. Zhou, "Image Super-Resolution with Non-Local Sparse Attention," 2021 IEEE / CVFConference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 2021, pp. 3516-3525. In the main body of the super-resolution module, non-local sparse attention blocks are inserted between lightweight residual blocks, thereby introducing sparse attention to prevent the network from focusing on noisy and less informative locations.

[0032] See Figure 1 This invention provides a single-stage low-resolution action recognition method based on super-resolution enhancement. Existing low-resolution action recognition methods based on super-resolution enhancement often operate in a two-stage manner, breaking the connection between super-resolution and action classification. This invention provides an end-to-end single-stage low-resolution video action recognition framework that integrates a super-resolution module and an action recognition module into a single framework. It requires only low-resolution data and does not rely on high-resolution data. Within a single stage, it is action-recognition oriented, jointly training video super-resolution and action classification. See also... Figure 2 The embodiments provide an improved lightweight and efficient super-resolution module. Lightweight residual blocks are designed for feature extraction, using two shuffle blocks instead of expensive convolution operations within each lightweight residual block. A ReLU activation layer is placed between the two shuffle blocks, and finally, the residual structure is achieved through element-wise addition with the input. This approach maintains feature extraction capability while controlling computational complexity. A preferred suggestion is to insert one non-local sparse attention block between every eight lightweight residual blocks, thereby introducing a sparse attention mechanism to prevent the network from concentrating on noisy and information-poor locations.

[0033] Specifically, the present invention provides a single-stage low-resolution action recognition method based on super-resolution enhancement, comprising the following steps:

[0034] Step 1: Construct a training sample set, which includes collecting human motion video data, downsampling the video data to a low resolution, and labeling the motion categories as tags. The low-resolution human motion video data and the corresponding motion category tags are then packaged into a low-resolution motion recognition dataset.

[0035] In practice, Python scripts can be used to downsample the original data, Vott software can be used to label the action categories, and a training sample set can be built in advance.

[0036] The embodiment includes the following sub-steps:

[0037] Step 1.1: Use various video recording terminals as data sources to acquire video data of human movements in real-world scenarios.

[0038] Step 1.2: Use a Python script to read the raw data, downsample it to a specified low resolution, and obtain low-resolution human motion video data.

[0039] Step 1.3: Open the motion video data with Vott software, add motion tags to each video, and finally save the txt file with the category tags.

[0040] Step 1.4: Package all the labeled low-resolution action recognition data into a training sample set.

[0041] Step 2: Set up a single-stage low-resolution action recognition framework based on super-resolution enhancement. Using low-resolution action video data as input, the data first passes through a super-resolution module to obtain super-resolution features, which are then input into an action classifier to output action categories. Finally, a comprehensive loss function is calculated to jointly optimize the entire framework. The super-resolution module uses an improved module designed in this embodiment of the invention, while the action classification module can be flexibly replaced with any other action classifier.

[0042] Step 3: Set up an improved, lightweight, and efficient super-resolution module, including mean transformation, preprocessing, main body, and upsampling. First, perform mean transformation and preprocessing on the input data to ensure similar statistical characteristics across different input data. Simultaneously, the preprocessing module uses larger convolutional kernels, allowing the network to cover a larger receptive field in the early stages. Second, stack multiple lightweight residual blocks and non-local coefficient attention blocks to increase the model size and improve performance. Finally, the upsampling module obtains super-resolution enhanced features.

[0043] The preferred implementation method provided in the embodiments includes the following sub-steps:

[0044] Step 3.1, Mean Transformation and Preprocessing. First, the input is standardized to conform to a standard normal distribution, which is beneficial for model training. Preferably, the preprocessing module consists of two residual structures, each containing two convolutional layers with 5×5 convolutional kernels.

[0045] Step 3.2: The main body consists of lightweight residual blocks and nonlocal sparse attention blocks. Preferably, eight lightweight residual blocks are inserted between two adjacent nonlocal attention blocks, for a total of three nonlocal sparse attention blocks and sixteen lightweight residual blocks.

[0046] Step 3.3: The upsampling part achieves image reconstruction through two convolutions and pixel cleaning.

[0047] Step 3.4: Finally, the super-resolution features with the original statistical properties of the image are output through mean transformation.

[0048] The above specific implementation scheme is a configuration that balances computational complexity and accuracy. Other similar super-resolution modules in other frameworks should also be within the scope of protection of this invention.

[0049] Step 4: Using the super-resolution features output by the super-resolution module as input, extract and classify the features in the action classifier to obtain the corresponding action category results. In the specific implementation of the action classification module, any action classifier can be flexibly adopted.

[0050] Step 5: Calculate the self-consistency, neighborhood consistency, and cross-entropy loss respectively, and train the entire model by combining the loss function through backpropagation.

[0051] A weighted comprehensive loss function is employed to fully leverage the advantages of combining super-resolution and action recognition within a single framework. Based on the Adam optimization algorithm, the model's output is guided to approximate the action category label information as closely as possible. Backpropagation iteratively solves for the network parameters that minimize the loss function, thereby obtaining a well-trained low-resolution action recognition model. Specifically, this includes the following sub-steps:

[0052] Step 5.1, calculate the self-consistency loss. The super-resolution features generated by the model should have a similar distribution to the low-resolution input image. Therefore, in this example, the super-resolution features are first downsampled using average pooling to obtain a feature map of the same size as the input image.

[0053]

[0054] in Indicates pooling operation, I LR and I SR These represent the low-resolution image and its super-resolution enhancement features, respectively. SRD Indicate ISR After pooling and I LR Versions of the same size. Implementation examples define self-consistency loss. as follows:

[0055]

[0056] Where ε is a constant.

[0057] Step 5.2, calculate the neighborhood consistency loss. The example further evaluates the quality of the enhanced image using the neighborhood consistency loss, preserving the spatial consistency of the super-resolution features by retaining the neighborhood differences between the input image and its enhanced version.

[0058]

[0059] Where K is the number of local regions, and Ω(i) represents the four adjacent regions (top, bottom, left, right) centered on region i. L represents the low-resolution image I. LR In the pixel value, S represents I SRD The pixel values ​​in the image.

[0060] Step 5.3, calculate the cross-entropy loss. For action recognition, we use the standard cross-entropy loss:

[0061]

[0062] Where y represents the action category predicted by the network. Provide the true label for the action category.

[0063] Step 5.4, calculate the comprehensive loss function. The example uses a linear combination of self-consistency loss, neighbor consistency loss, and cross-entropy loss for end-to-end optimization throughout the network. In summary, the example proposes using the following comprehensive loss function for training:

[0064]

[0065] α is used to determine the scale of super-resolution feature quality, and β is used to balance neighborhood consistency.

[0066] Step 6: Based on the low-resolution action recognition model trained in Step 5, input the low-resolution video human action data to be recognized, and perform action recognition on the test data.

[0067] In specific implementation, the method proposed in the technical solution of this invention can be automatically executed by those skilled in the art using computer software technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the protection scope of this invention.

[0068] In some possible embodiments, a computer device / apparatus / system is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described single-stage low-resolution action recognition method based on super-resolution enhancement.

[0069] In some possible embodiments, a computer-readable storage medium is provided having a computer program / instructions stored thereon, which, when executed by a processor, implement the steps of the above-described single-stage low-resolution action recognition method based on super-resolution enhancement.

[0070] In some possible embodiments, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described single-stage low-resolution action recognition method based on super-resolution enhancement.

[0071] The specific examples described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific examples or use similar methods to replace them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A single-stage low-resolution action recognition method based on super-resolution enhancement, characterized in that: An end-to-end unified single-stage low-resolution video action recognition framework is established for action recognition. This framework integrates a super-resolution module and an action recognition module, and is action-oriented within a single stage, requiring only low-resolution data and not relying on high-resolution data, and jointly training the super-resolution and action classification modules. From the perspective of no-reference image quality evaluation, spatial consistency is maintained by evaluating the changes in self-consistency and neighborhood consistency of the input and output features of the super-resolution module. A comprehensive loss function consisting of a linear combination of self-consistency loss, neighborhood consistency loss, and cross-entropy loss is set, and end-to-end optimization is achieved through backpropagation throughout the framework. In the super-resolution module, a lightweight residual block is set for feature extraction, and a non-local attention module that combines sparsity is embedded into the main body of the super-resolution module. When implementing action recognition based on the end-to-end unified single-stage low-resolution video action recognition framework, low-resolution action video data is used as input. First, the super-resolution module obtains super-resolution features, which are then input into the action classifier to output the action category. Subsequently, a comprehensive loss function is calculated to jointly optimize the entire framework. The self-consistency loss as follows: in It is a constant. and These represent low-resolution images and super-resolution enhancement features, respectively. Spatial consistency as follows: in It is the number of local regions. Indicated by region The center is divided into four adjacent areas: above, below, left, and right. Represents low-resolution images Pixel values ​​in express The pixel values ​​in the image.

2. The single-stage low-resolution action recognition method based on super-resolution enhancement according to claim 1, characterized in that: The lightweight residual block contains two Shuffle blocks. A ReLU activation layer is set between the first and second Shuffle blocks. After the second Shuffle block, the input of the lightweight residual block is added element-wise to achieve the residual structure.

3. The single-stage low-resolution action recognition method based on super-resolution enhancement according to claim 2, characterized in that: The non-local attention module that incorporates sparsity is embedded into the main body of the super-resolution module. This is achieved by inserting non-local sparse attention blocks between lightweight residual blocks, thereby introducing sparse attention to prevent the network from concentrating on noisy and less information-rich locations.

4. The single-stage low-resolution action recognition method based on super-resolution enhancement according to claim 3, characterized in that: Insert one nonlocal sparse attention block between every eight lightweight residual blocks.

5. A computer system comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the single-stage low-resolution action recognition method based on super-resolution enhancement as described in any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that: When the computer program or instructions are executed by a processor, they implement the steps of the single-stage low-resolution action recognition method based on super-resolution enhancement as described in any one of claims 1 to 4.

7. A computer program product, comprising a computer program or instructions, characterized in that: When the computer program or instructions are executed by a processor, they implement the steps of the single-stage low-resolution action recognition method based on super-resolution enhancement as described in any one of claims 1 to 4.

Images